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Introduction 

Robotic  tele-surgery  is  a  promising  application  of  robotics  to  medicine,  aiming  to  enhance  the 
dexterity  and  sensation  of  minimally  invasive  surgery  through  use  of  millimeter-scale  robotic 
manipulators  under  the  control  of  a  surgeon.  Current  interfaces  prevent  surgeons  from 
exploiting  sensory  capabilities,  leading  to  underutilization  of  many  potential  capacities  offered  by 
robotic  tele-surgery.  To  advance  the  state  of  the  art  in  the  field  of  human-machine  interface 
(HMI)  design  for  tele-robotic  surgery,  the  Florida  Institute  for  Human  and  Machine  Cognition 
(IHMC)  executed  the  Anthro-Centric  Multisensory  Interface  for  Sensory  Augmentation  of  Tele- 
Surgery  (ACMI-SATS)  project  for  the  U.  S.  Army  Telemedicine  and  Advanced  Technology 
Research  Center  (TATRC).  ACMI-SATS  developed  methods  to  improve  tele-surgery 
effectiveness  in  military  and  civilian  applications  by  improving  the  sensory  and  control  interface 
using  HMI  principles  to  increase  efficiency  and  reduce  surgeon  cognitive  workload  and  errors. 

Background 

Tele-operated  surgical  (tele-surgical)  endoscopic  robotic  systems  have  rapidly  become  the 
standard  of  practice  for  many  surgical  procedures.  This  technology  reduces  surgical  trauma  to 
the  patient,  lowers  postoperative  complications,  improves  surgeon  ergonomics,  reduces  effects 
of  surgeon  motion  tremors  and  enables  the  performance  of  novel  techniques  (e.g.,  the  end 
effector  can  perform  actions  that  a  human  appendage  cannot,  such  as  locking  in  position  or 
rotating  past  anatomical  limits).  These  “surgery  by  wire”  systems  can  remove  the  need  to 
transport  the  patient  or  place  the  surgeon  in  harm's  way  to  provide  definitive  surgical  care  and 
have  the  potential  to  proactively  prevent  undesirable  actions,  automate  repetitive  tasks  and 
enable  safer  training  through  simulation.  Hirshauer  (1991),  however,  notes  that  endoscopy 
separates  the  eye  from  the  hand  of  the  surgeon,  preventing  perception  of  the  useful  kinesthetic 
feedback  from  direct  interaction  with  the  tissues  present  in  traditional  “open”  procedures,  and 
the  reliance  on  video  displays  creates  a  visual  information  flow  bottleneck.  This  increases  the 
visual  workload  and  can  lead  to  increased  surgical  errors,  longer  intraoperative  sessions  (with 
accompanying  increased  anesthesia  risks)  and  decreased  number  of  procedures  performed  in  a 
given  time  period.  Technological  systems  that  utilize  visually  dominant  displays,  such  as  those 
used  in  virtual  environment  (VE)  simulation  and  training,  have  shown  improved  user  task 
performance  when  combining  both  tactile  and  visual  sensory  information  (Briggs  &  Srinivisan, 
2001;  He  &  Agah,  2001).  Virtual  reality  (VR)  training  systems  have  been  developed  to  bring  VE 
simulation  to  surgical  training  (Satava,  1995;  Gallagher  et  al.,  2005),  however,  VEs  that  lack 
realistic  kinesthetic  and  haptic  feedback  show  increased  errors  (Cohn  et  al.,  2000).  Combining 
visual,  spatial  audio  and  tactile  sensory  modalities,  can  improve  performance  and  situation 
awareness  (SA)  in  complex  dynamic  tasks  (Raj,  Kass  &  Perry,  2000;  Diamond  et  al.,  2003; 
Olson  et  al.,  2004;  Erikson  et  al.,  2008;  Fuchs  et  al.,  2008;  Merlo,  Gilson  &  Hancock,  2008). 

Telerobotic  systems  such  as  the  da  Vinci®  Si  HD  (Intuitive  Surgical,  Sunnyvale,  CA)  must 
employ  torque,  position,  velocity  and  strain  sensors  to  maintain  accurate  closed-loop, 
servomotion.  Beyond  control  loop  error  based  force-feedback,  however,  presentation  of  these 
data  streams  in  raw  form  would  not  provide  the  requisite  sensory  data  needed  to  enhance  the 
surgeon’s  experience.  The  addition  of  high  definition  (HD)  stereoscopic  visual  displays  can 
provide  sufficient  cues  to  create  a  visual  perception  of  haptic  feedback  (Hagen  et  al.,  2008).  A 
sensory  experience  limited  to  vision,  however,  still  lacks  the  richness  and  subtleties  available 
during  open  surgery  and  decrements  procedural  consistency,  workload  and  operative  efficiency 
(Tavakoli,  Patel  &  Moallem,  2005;  Zhou  et  al.,  2008).  Moreover,  these  systems  require  that  the 
surgeon  remain  seated,  with  his  or  her  head  down  at  a  control  console  to  perceive  the  three 
dimensional  video  and  use  the  fixed  hand  controllers.  While  this  compares  favorably  to  standing 
astride  an  operating  table  using  laparoscopic  instruments,  the  most  favorable  ergonomics  would 
facilitate  free  movement,  posture  and  position  to  reduce  fatigue  and  potentially  allow  the 
surgeon  to  perform  more  procedures  in  a  given  time. 
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In  the  military  medicine  domain,  tele-surgical  systems  have  the  potential  to  revolutionize  combat 
casualty  care  by  bringing  definitive  surgical  trauma  care  to  the  wounded  servicemember  without 
placing  surgical  specialists  in  harm’s  way.  Current  technology  developments  such  as  the  United 
States  Department  of  Defense  (DoD)  Defense  Advanced  Research  Projects  Agency  (DARPA) 
Trauma  Pod  (Figure  1)  seek  to  further  reduce  the  time  needed  for  delivery  of  surgical 
intervention  by  installing  a  complete  tele-surgical  system  into  robotic  vehicles  (Burnett,  2007; 
Satava,  2005a,  2005b).  These  Trauma  Pods  could  be  remotely  driven  to  wounded  soldiers  in 
the  field,  providing  an  opportunity  to  treat  severe  wounds  within  the  “golden  hour”  following 
injury  (Pueschel,  2006).  Deploying  a  fleet  of  such  vehicles  to  a  theater  of  operations  would 
provide  maximum  benefits  by  providing  the  potential  for  immediate  care  to  multiple  wounded 
servicemembers  with  polytraumatic  injuries  following,  for  example,  a  detonation  of  an 
improvised  explosive  device  (IED).  In  such  a  scenario,  if  a  small  team  of  surgical  specialists 
(e.g.,  orthopedic,  cardiovascular,  neuro,  general/trauma  surgeons,  etc.)  could  easily  and  safely 
handoff  control  of  the  robotic  system  between  each  other  as  needed,  they  could  potentially 
stabilize  and  provide  definitive  care  to  the  maximum  number  of  casualties.  Because  there  would 
be  no  cross  contamination  between  patients  (or  need  to  scrub  or  change  sterile  garments),  the 
trauma  team  could  manage  the  surgical  needs  of  multiple  patients  more  effectively.  Experience 
with  other  robotic  systems,  such  as  unmanned  aerial  vehicle  (UAVs),  however,  has  shown  that 
the  addition  of  robotic  technology  to  a  given  mission  can  actually  adversely  increase  manpower 
requirements  (Franke,  et  al.,  2005). 


Figure  1:  DARPA  Trauma  Pod  Concept  (lead  by  SRI  International)  loading  a  wounded  servicemember 
(left),  evaluating  the  trauma  (center)  and  performing  surgery  (right).  Images  downloaded  from: 
http://www.technovelav.com/ct/Science-Fiction-News. asp?NewsNum=364.  and 

http://brl.ee.washington.edu/Research  Active/Surqerv/Proiect  09/Proiect  09.html 


The  addition  of  HD  stereoscopic  visual  displays  to  robotic  surgery  systems  provides  an  initial 
step  toward  improved  sensory  interface  design.  However,  lack  of  haptic  feedback  in  endoscopy 
can  lead  to  surgical  errors  such  as  tearing  of  tissues,  broken  sutures  and  hemorrhage, 
especially  in  delicate  or  intricate  procedures  (Bethea  et  al.,  2004;  Vassiliades,  2006).  One 
approach  to  enhancing  haptic  feedback  embeds  tactile  force  reflection  transducers  on  the 
fingertips  in  the  surgeon  console  hand  controllers  (Figure  2)  to  provide  a  reflection  of  the  force 
applied  by  endoscopic  graspers  capabilities  (Morimoto  et  al.,  1997;  Bar-Cohen  et  al.,  2001; 
Tavakoli,  Patel,  &  Moallam,  2005;  Culjat  et  al.,  2008).  While  this  technique  has  demonstrated 
improved  surgical  technique  (reduced  grasping  force),  the  surgeon’s  auditory  and  tactile 
sensory  capabilities  remain  underutilized  while  his  or  her  visual  sense  remains  highly  task 
loaded.  Alternate  haptic  interfaces  that  provide  end  effector  force  feedback  indicate  proportional 
resistance  to  commanded  movements  and  complement  the  visual  display  to  reduce  workload 
(Wagner,  Stylopulos,  Jackson  &  Howe,  2007).  However,  the  narrow  field  of  view  provided  by  the 
endoscope  negatively  impacts  situation  awareness  (SA)  by  limiting  perception  of  complex 
spatial  relationships  within  the  body  cavity.  This  loss  of  SA  may  lead  to  inadvertent  incursion 
into  sensitive  structures  following  camera  orientation  changes  or  movement  of  internal  organs. 
These  haptic  approaches  borrow  concepts  from  the  VE  and  VR  worlds  that  place  a  high 
premium  on  presence  (the  convincing  sense  of  perceiving  as  if  physically  in  the  VR  or  VE).  In 
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contrast,  tele-surgery  must  place  the  highest  premium  on  effective  treatment  of  the  patient  by 
optimizing  perception  of  patient  state  to  the  surgeon.  Because  the  feedback  generated  by  the 
sensors  in  the  operative  field  is  no  longer  directly  coupled  to  the  surgeon’s  hands  (either 
through  direct  contact  with  the  tissues  or  indirect  contact  via  rigid  laparoscopic  instruments), 
opportunities  now  exist  to  utilize  other  sensory  substrates.  In  fact,  some  actions  in  surgical 
robotic  manipulation  can  cause  the  surgeon  to  make  inadvertent  hand  movements  due  to  haptic 
force  feedback  errors  (Tavakoli,  Patel,  &  Moallem,  2005).  Unlike  classic  VE/VR  approaches  to 
haptics  in  tele-surgery,  these  alternative  display  technologies  could  complement  or  supplant 
traditional  haptics  without  affecting  or  interfering  with  the  motion  of  the  primary  control  input 
mechanism.  For  this  discussion,  we  define  haptics  as  telepresence  related  kinesthetic  sensory 
interactions  and  tactile  as  general  or  abstract  tactual  interactions. 


Figure  2:  (left)  Electro-rheological  fluid  (ERF)  based  haptic  surgeon  hand  interface  (Bar-Cohen  et  al., 
2001);  (center)  Force  transducer  fitted  to  endoscopic  grasper;  (right)  Pneumatic  haptic  finger  tip  display 
(Culjat  et  al.,  2008)  integrated  with  da  Vinci®  hand  controllers. 

Because  perception  takes  place  in  the  brain,  not  at  the  end  organ  (Bach-y-Rita,  1972),  the  brain 
can  learn  to  reinterpret  the  meaning  of  signals  from  specific  nerves  (e.g.,  from  tactile  receptors) 
given  appropriate  self-generated  feedback  (Buonomano  &  Merzenich,  1998).  This  forms  the 
basis  for  interfaces  that  can  non-invasively  and  unobtrusively  use  alternative  sensory  pathways 
to  provide  information.  This  means  that  the  information  displayed  does  not  necessarily  need  to 
represent  the  underlying  data  at  high  resolution,  rather  abstract  representations  of  the  sensory 
environment  information  can  provide  sufficient  data  for  operator  decision  making  and  improved 
SA  (Raj,  Kass  &  Perry,  2000).  Such  sensory  substitution  mechanisms  have  been  demonstrated 
in  tele-surgical  applications  (Kitagawa  et  al.,  2005)  and  exploit  the  plasticity  inherent  to  the  brain 
and  nervous  system,  which  supports  both  long-term  and  short-term  anatomical  and  functional 
remapping  of  sensory  data  (Finkel,  1990;  Walcott  &  Langdon,  2001).  Therefore,  a  new  class  of 
interfaces  could  be  incorporated  into  the  tele-surgical  surgeon  interface.  Tactile  and  spatial 
audio  displays  that  provide  higher-level  representations  of  data  could  provide  metadata  or 
information  that  a  surgeon  could  not  physically  sense  directly,  even  during  an  open  procedure. 

Divorcing  the  surgeon’s  sensory  perception  from  direct  mechanical  linkage  to  the  patient,  the 
operative  field,  and  the  surgical  instruments  could  also  enable  a  paradigm  shift  in  tele-surgical 
methods.  Laparoscopic  instruments  were  initially  developed  as  extended  versions  of  their 
counterparts  used  in  open  procedures  (e.g.,  extended  length  scissors,  needle  drivers,  etc.); 
even  though  the  scaling  induced  poor  ergonomics,  familiarity  with  the  older  tools  existed. 
Likewise,  current  tele-surgery  systems  utilize  tools  that  mimic  the  laparoscopic  instruments, 
replacing  the  direct  mechanical  connection  with  an  electronic  one.  The  functional  capabilities 
(e.g.,  degrees  of  freedom)  of  the  robot  end  effectors  (such  as  the  da  Vinci®  EndoWrist®)  exceed 
those  of  laparoscopic  instruments  and  human  hands.  Advances  in  head  mounted  displays 
(HMDs)  and  motion  capture  from  the  commercial  sector  have  advanced  sufficiently  to  allow  both 
freedom  of  movement  and  high-resolution  stereoscopic  visual  perception  without  confining  the 
user  to  a  fixed  console.  Motion  capture  could  translate  movements  that  match  the  scale  and 
trajectories  of  normal  (real-world)  kinesthetic  interactions,  as  well  as  enable  the  use  of  motion 
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gestures  to  accomplish  supervisory-controlled  actions  to  take  advantage  of  the  capabilities  of 
modern  robotic  surgical  systems.  Because  kinesthetic  perception  includes  both  self-perception 
(proprioception)  and  perception  of  changes  in  the  environment  (Hannford  &  Venema,  1995), 
proprioceptive  sensations  could  disambiguate  abstract  sensory  substitution  displays  of  end 
effector  motion.  The  full  use  of  arm  movements  (versus  hands  only)  could  provide  more 
accurate  mapping  to  the  degrees  of  freedom  available  in  modern  surgical  robots. 

Clearly,  adding  additional  sensory 
channels  would  increase  the  volume  of 
information  flow  to  the  surgeon.  The 
human  brain,  however,  has  developed  to 
make  use  of  multi-sensory  inputs  in  the 
natural  world,  which  provides  a  richer 
sensory  environment  than  most  man¬ 
made  systems.  This  increase  in 
information  transduction  will  also 
markedly  increase  the  bandwidth 
requirements  for  data  flow  between  the 
robot  (e.g.,  installed  in  a  Trauma  Pod) 
and  the  remote  surgeon  (Hannford  & 

Venema,  1995).  Recent  DoD  projects 
such  as  the  High  Altitude  Platforms 
Mobile  Robotic  Tele-surgery  (HAPsMRT) 

(Broderick,  2006;  Rosen  &  Hannaford, 

2006)  and  the  Smart  Codec  with  Tele¬ 
surgery  Capability  (Energid  Technologies, 

Cambridge,  MA),  respectively,  have 
developed  methods  that  use  unmanned 
aerial  vehicles  to  reduce  the  radio 
frequency  signal  lag  between  the  surgeon 
and  the  robotic  end  effectors  and  to 
develop  efficient  algorithms  for  data 
transmission  (Figure  3). 

When  processing  and  interpreting  tactile  data,  specific  traits  of  human  sensation  and  cognitive 
processing  (such  as  adaptation,  habituation,  or  satiation  to  durative  stimuli)  may  interfere  with 
perception  of  persistent  tactile  and  other  sensory  stimuli.  Adaptation  occurs  when  a  signal 
persists  for  an  extended  period  of  time  (e.g.,  the  tactile  sensation  of  wearing  a  watch  is 
automatically  filtered  out  by  the  peripheral  and  central  nervous  system).  Habituation  occurs 
when  a  durative  signal  repeats  periodically  (e.g.,  a  ticking  clock)  and  is  no  longer  perceived. 
Satiation,  also  the  product  of  prolonged  stimulation,  produces  specific  spatial  distortions 
(Cholewiak,  1976),  perhaps  based  on  transient  plasticity  (Whitsel  et  al.,  1989;  Merzenich, 
Recanzone,  Jenkins,  Allard  &  Nudo,  1988).  These  are  issues  with  known  solutions  currently 
implemented  in  IHMC's  Adaptive  Multiagent  Integration  (AMI)  software  (e.g.,  modulation  of 
waveform  characteristics  of  persistent  cues)  that  mitigates  their  effects.  AMI  agents  can  also 
generate  sensory  illusions,  such  as  spatial  audio,  the  saltation  rabbit  illusion  (Geldard  & 
Sherrick,  1972,  Geldard,  1975;  Bremer,  Pittenger,  Warren,  &  Jenkins,  1977)  and  spatial 
summation  (Craig,  1968).  Recent  studies  have  confirmed  that  tactile  illusions  are  processed  at 
the  cortical  level  as  if  they  were  real;  that  is,  no  differences  between  illusory  and  veridical  tactile 
stimulation  patterns  manifest  in  functional  magnetic  resonance  (fMRI)  scans  of  the  human  brain 
(Blankenburg,  Ruff,  Deichmann,  Rees  &  Driver,  2006).  Therefore,  tactile  illusions  could  improve 
perception  of  tactile  signals  beyond  the  display's  physical  resolution. 


Figure  3:  High  Altitude  Platforms  Mobile  Robotic  Tele¬ 
surgery  (HAPsMRT)  technology  demonstration  using 
real  time  data  compression  and  UAV  relay  for  haptic 
tele-surgery  interactions  (Broderick,  2006).  A  UAV  (left, 
upper)  relays  surgeon  movements  (left,  lower)  to  a 
surgical  robot  (right,  upper).  Haptic  hand  controllers 
provide  feedback  for  suture  task  (right,  lower). 
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Prior  research  by  this  team  and  others  on  tactile  sensory  substitution  has  shown  that  the  brain  is 
capable  of  integrating  the  information  from  an  artificial  receptor,  arriving  via  a  brain-machine 
interface  (BMI),  in  a  perceptual  experience  that  depends  upon  the  nature  of  the  information 
gathered  by  the  specific  artificial  receptor  (Bach-y-Rita,  1972,  1995;  Bach-y-Rita,  Collins, 
Saunders,  White  &  Scadden,  1969).  The  BrainPort®  device  (Wicab,  Inc.,  Madison,  Wl)  provides 
information  to  the  brain  through  electrotactile  stimulation  of  the  tongue  (Bach-y-Rita, 
Kaczmarek,  Tyler,  &  Garcia-Lara,  1998).  Receptors  on  the  tongue  surface  are  uniquely  qualified 
to  receive  electrical  impulses  because  of  the  density  and  sensitivity  of  nerve  fibers  at  this  site, 
the  electrolyte  (saliva)  that  allows  it  to  receive  and  maintain  electrical  contacts  and  its  proximity 
to  the  brainstem,  minimizing  perceptual  lag.  With  the  highest  resolution  device  currently 
available,  an  approximately  25x25  tactile  image  (some  are  not  active)  is  created  by  a  sequence 
of  pulses  presented  at  a  rate  of  200  Hz.  The  amplitude  value  (voltage)  of  the  pulse  sequence  or 
'burst',  updated  at  50  Hz,  varies  with  the  grayscale  level  of  a  video  image.  The  participant 
controls  the  overall  intensity  (voltage)  level,  which  does  not  exceed  15VDC.  The  second  novel 
electrotactile  interface,  the  VideoTact,  which  is  placed  on  the  abdomen  and  worn  discretely 
under  the  user’s  clothing,  exploits  the  larger  surface  area  of  the  abdomen  to  provide  a  larger 
display  area.  While  the  density  of  torso  sensory  receptors  is  lower  than  the  tongue,  placing  a 
high-resolution  display  (i.e.,  24x32)  on  the  abdomen  allows  for  rapid  perception  of  object  motion 
(Bach-y-Rita  et  al.,  1969).  An  electrolyte  gel  ensures  electrical  contact  with  the  skin. 


Figure  4:  Non-visual  interfaces  utilized  in  ACMI-SATS:  (left)  BrainPort  Intraoral  Device  electrotactile 
tongue  array  (600  tactile  pixels),  (center  left)  VideoTact  (ForeThought  Development,  LLC,  Blue  Mounds, 
Wl)  electrotactile  abdominal  display  (768  tactile  pixels),  (center  right)  Tactile  Situation  Awareness  System 
(TSAS)  implementation  with  24  vibrotactile  transducers  in  a  tactile  torso  interface  (TTI).  (right)  10.2 
speaker  surround  sound  system. 


A  number  of  recent  and  ongoing  studies  with  the  have  shown  that  blind  individuals  can 
recognize  elements  of  standard  eye  charts  (Sampaio,  Maris  &  Bach-y-Rita,  2001,  Kupers, 
Sampaio,  Moesgaard,  Gjedde  &  Ptito,  2003)  and  labyrinthine  defective  individuals  (those  with 
inner  ear  balance  disturbances)  can  dramatically  improve  balance  (Tyler,  Danilov  &  Bach-y- 
Rita,  2003)  by  using  the  BrainPort®.This  requires  that  the  user  have  control  of  the  movements  of 
the  sensor  in  order  to  self-generate  feedback  to  reinforce  the  novel  sensory  stimuli.  In  the 
absence  of  the  sensory-motor  loop,  we  have  observed  that  the  sensory  stimulation  through  a 
tactile  array  is  perceived  as  a  purely  tactile  experience  localized  to  the  body  part  in  contact  with 
the  array.  Laboratory  studies  and  operational  test  and  evaluation  of  the  Tactile  Situation 
Awareness  System  (TSAS)  and  other  tactile  displays  have  shown  that  individuals  rapidly 
localize  and  respond  to  encoded  tactile  cues  presented  on  the  torso  and  limbs  (Calhoun, 
Draper,  Ruff  &  Fontejon,  2002;  Diamond,  Kass,  Andrasik,  Raj  &  Rupert,  2003;  Raj,  Kass,  & 
Perry,  2000;  Sarter,  2001;  McGrath,  Estrada,  Braithwaite,  Raj  &  Rupert,  2004).  This  forms  the 
rationale  for  utilizing  self-generated  natural  movements  of  the  hands,  arms  and  body  to  control 
robotic  systems.  The  current  IHMC  TSAS  implementation  uses  a  vest  with  24  tactors  in  a  tactile 
torso  interface  (TTI)  that  can  operate  effectively  with  the  VideoTact  and  the  BrainPort®  devices. 


Natural  motion  gesture-based  control  reduces  the  workload  and  learning  associated  with 
standard  joystick/button  input  devices,  and  providing  low  latency  feedback  through  haptic 
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interfaces  allows  precise  control  of  an  external  system  with  minimal  training  time.  Blinded 
servicemembers  learned  to  stabilize  and  manipulate  the  tactile  image  with  their  prosthetic  eyes 
within  10-20  minutes,  demonstrating  that  the  tactile  feedback  does  not  need  to  be  presented 


through  the  hands  for  accurate  perception  and 
control  of  a  dynamic  action.  During  this  22- 
month  project,  we  evaluated  performance  of 
specific  tasks  using  the  Mimic  training  simulator 
and  the  Zeus,  using  gestural  commands  with 
tactile  and  auditory  augmentation  of  large  screen 
visual  displays.  In  this  project,  we  have  used 
gesture  recognition  agents  in  AMI,  based  on  a 
finite  state  machine  (FSM)  approach  to 
recognize  gestural  requests  to  lock  robot  joints 
during  specific  tasks,  such  as  selecting  the  robot 
arm,  switching  between  simultaneous 
procedures,  and  handing-off  control  between  the 
surgeon  and  first  assistant.  We  have 
demonstrated  that  one  user  can  switch  at  will 
between  two  surgical  procedures  and  that  two 
participants  can  share  control  of  the  same  robot, 
using  head  tracking  and  natural  movement 
gestures  as  an  efficient  method  of  hand-off  or 
procedure  switching. 


Figure  5:  Body  position  (inset)  tracked  with 
ShapeWrap  system  using  IHMC’s  AMI 
architecture.  Box  projected  under  avatar 
indicates  stable  platform  for  a  given  stance. 


Obtaining  high  gesture  recognition  accuracy  requires  the  ability  to  accurately  sense  the  position 
and  configuration  of  a  user’s  hands,  limbs,  head  and  torso.  IHMC  has  developed  custom 
software  implemented  in  AMI  that  allows  rapid,  accurate  detection  (within  a  few  millimeters)  of 
relative  positions  of  body  components,  estimation  of  center  of  gravity,  ground  reaction  force, 
center  of  pressure  and  estimation  of  dynamic  stability.  This  system  incorporates  data  from  a 
custom  built  wireless,  wearable  force  sensing  insole  system  with  132  load  cells  per  foot 
(Pressure  Profile  Systems,  Los  Angeles,  CA)  and  a  full  body/limb/head  wireless,  wearable 
motion  capture  system  (ShapeWraplll,  Measurand,  Inc,  Fredericton,  NB)  that  has  been 
integrated  into  the  ACMI-SATS  prototype  to  provide  the  user  with  sense  of  proprioception  and 
interaction  with  the  ground  through  the  haptic  interfaces  (Figure  5). 


Figure  6:  (left)  Gesture  based  controller  for  robotic  arm  simulator,  (right)  Status  post  binocular 
enucleation,  a  servicemember  uses  his  prosthetic  eyes  to  control  a  tracking  reticle  on  the  screen  with 
tactile  (tongue  based)  display  of  area  displayed  inside  the  reticle  (near  user’s  finger). 
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In  addition  to  the  loss  of  qualitative  haptic  and  tactile  sensation  rising  from  the  transition  from 
open  to  endoscopic  procedures,  the  surgeon  also  loses  the  significant  kinesthetic  feedback 
associated  with  somatic  movement.  Through  motion  scaling  in  robotic  systems  (unlike 
laparoscopic  systems)  surgical  robots  can  translate  gross  arm,  torso  and  head  movement 
gestures  into  precise  movement  of  surgical  instruments.  These  large  movements  can  provide  a 
kinesthetically  rich  experience  for  the  user,  who  can  move  freely  and  with  natural  motion. 
Previously,  IHMC  and  its  research  partners  demonstrated  gestural  control  of  fine  movements  of 
a  robotic  arm  simulator  and  stabilization  of  visual  imagery  represented  tactually  to  recently 
blinded  servicemembers.  In  the  former  application,  the  operator  controls  the  National 
Aeronautics  and  Space  Administration  (NASA)  Basic  Operational  Robotics  Instruction  System 
(BORIS)  software  robot  (Gilliland  et  al.,  2005)  with  natural  arm  and  hand  movements.  In  the 
latter,  the  BrainPort®  tactually  presented  image  provides  "gaze"  movement  cues  to  the  user  to 
dynamically  position  his  extraocular  muscle  controlled  prosthetic  (glass)  eyes  (Figure  6). 

Project  Activities 

While  awaiting  delivery  of  our  da  Vinci®  S  HD  simulator  (dV-Trainer,  Mimic  Technologies,  Inc., 
Redmond,  WA),  we  initiated  development  of  a  simulation  environment  for  IHMC's  existing  Zeus 
surgical  robot  hardware  (Computer  Motion,  Goleta,  CA),  and  assembled  a  simulation  area  in  our 
laboratory  space.  We  acquired  two  additional  Aesop  arms  that  are  functionally  equivalent  to 
Zeus  arms  (aside  from  the  micro  wrist)  to  implement  a  four-arm  solution  (Figure  7)  similar  to  the 
da  Vinci®  system  for  testing  hardware  modifications  prior  to  implementation  on  the  Zeus. 


Figure  7:  (left,  top)  IHMC  ACMI-SATS  Zeus  configuration  with  four  robot  arms.  In  the  center  of  the 
operating  table,  a  plexiglass  “patient  simulator”  provides  a  closed  environment  for  endoscopic  activities, 
(left,  lower)  The  Zeus  can  be  controlled  by  the  stock  control  input  devices,  or  with  more  natural 
movements  using  motion  capture,  (right)  Human  scale  projected  imagery  from  the  Zeus  testbed. 
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In  parallel  with  circuit  mapping  of  the  Zeus  controllers  to  extract  position  and  force  sate 
variables  from  the  Zeus  actuators,  we  developed  a  virtual  prototype  of  the  Zeus  robot  in  order  to 
test  hypotheses  without  the  constraints  of  interaction  with  physical  hardware.  We  implemented 
the  virtual  robot  in  the  Java  programming  language  using  Simulation  Construction  Set  (SCS) 
from  Yobotics,  Inc  (Cincinati,  OH).  The  simulator  created  an  accessible  environment  that  allows 
for  rapid  evaluation  and  testing  of  tele-surgery  augmentation  concepts.  The  simulation  (Figure 
7)  provides  a  physics-based,  kinematically  accurate  graphical  simulation  of  the  robot  arms  and 
allows  for  development,  test  and  evaluation  of  ACMI-SATS  concepts  prior  to  testing  on  the 
hardware.  The  Yobotics  SCS  allows  both  simulated  input  and  real  world  sensor  inputs  to  drive 
the  motion  of  the  arms.  Input  signals  and  robot  responses  can  be  displayed  in  strip  chart  form 
and  recorded.  These  recordings  can  then  be  played  back  at  various  speeds  relative  to  real-time 
to  fully  explore  response  characteristics. 


Using  AMI,  we  integrated  our  wireless  full-body,  wearable  motion  capture  system 
(ShapeWraplll)  with  our  Yobotics  simulation  of  the  four-arm  Zeus.  We  updated  1970's  era  pen- 
plotter  routines  for  “look  ahead”  operations  that  use  the  system  model  to  smooth  motion  despite 
varying  time  steps  and  noise.  This  algorithm  "chops"  the  command  signal  at  a  high  frequency 
and  sends  these  small  time-step  data  packets  serially  to  the  actuators.  The  simulation  injects  a 
variable  delay  and  noise  to  the  signal.  At  the  actuator  end,  a  model  of  the  system  with  the 
known  sample  rate  helps  determine  signal  delays  and  extrapolate  the  likely  trajectory  when 
delays  occur.  This  feature  would  be  useful  for  remote  tele-surgical  applications  that  suffer  from 
intermittent  communications  or  low  bandwidth  (e.g.,  TraumaPod). 
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Figure  8:  (left)  Yobotics  Simulation  Construction  Set  implementation  of  IHMC’s  current  Zeus-based 
ACMI-SATS  surgical  robot  configuration.  Top  frame  shows  graphical  model  of  configuration  shown  in 
Figure  1.  Left  panel  provides  a  list  of  joints  in  the  system  and  their  orientations,  torques,  rates,  etc.  Lower 
panel  shows  strip  chart  representation  of  variables  selected  from  the  list  in  the  left  inset  panel,  (right, 
upper  panel)  Simplified  physics  only  model  of  two  robotic  arms  with  port  reaction  forces  modeled,  (right, 
lower  panel)  Velocity  command  trajectory  (red)  of  one  arm  "chopped"  at  high  frequency  (blue)  with  added 
(simulated)  noise  and  variable  delay  (green)  are  shown  in  the  top  plot.  The  middle  plot  shows  the  arm 
trajectory  (red)  reconstructed  piecewise  from  the  high  frequency  data  segments. 
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Figure  8  (right)  shows  an  intermittent  command  signal  (blue)  being  smoothed  in  to  a  smooth 
velocity  using  the  look-ahead  algorithm  (New  Scientist,  1972).  A  musical  instrument  digital 
interface  (MIDI)  control  surface  (BCF2000,  Behringer  International  GmbH,  Willich,  Germany) 
allows  for  manual  positioning  of  simulation  parameters  (such  as  friction,  rates  and  joint 
positions).  This  mechanism  allows  offset  adjustment  to  match  body  movements  to  robot  actions. 

We  modified  the  actual  Zeus  hardware  to  connect  to  a  computer  via  USB  using  analog  to  digital 
converters  (ADCs)  and  digital  to  analog  converters  (DACs)  to  allow  direct  control  of  the  robot 
arms  and  monitoring  of  both  position  and  current  draw  via  AMI  (USB-AI16-16A  and  USB-AI016- 
16A,  respectively,  Acces  I/O  Products,  Inc.,  San  Diego,  CA).  With  this  connection,  we  can  utilize 
auditory  and  tactile  interfaces  for  representation  of  Zeus  data  using  the  AMI  software  agents 
developed  with  the  Yobotics  simulation.  Using  these  agents,  we  developed  a  number  of 
electrotactile  symbologies  for  representation  of  the  positions  and  forces  on  the  end  effectors  as 
well  as  relative  locations  and  forces  on  the  robot  arm  shafts.  The  end  effector  information 
displays  on  the  tongue  using  the  BrainPort®  device  and  shaft  data  displays  on  the  abdomen  via 
the  VideoTact  electrotactile  interface.  Both  the  BrainPort®  and  the  VideoTact  devices,  as  well 
as  a  10.2  channel  surround  sound  system,  were  evaluated  as  potential  sensory  augmentation 
methods  for  surgical  interfaces.  We  transitioned  these  methods  to  the  Mimic  dv-Trainer  to 
include  direct  control  of  the  Mimic  dV-Trainer  actuator  positions,  orientations  and  graspers.  By 
presenting  the  visual  imagery  from  the  endoscope  or  the  simulation  onto  a  large  screen  (via 
projection  of  large  screen  LCD),  ACMI-SATS  maps  the  apparent  size  and  movement  of  the  end 
effectors  to  an  approximate  1:1  ratio  with  the  user's  arm  and  hand  movements  (Figure  9). 


Figure  9:  Mimic  dV-Trainer  imagery  projected  a  reasonable  approximation  of  the  size  of  the  human  hand 
and  arm.  Motion  capture  of  hand  and  arm  movements  maps  1:1  to  onscreen  robot  movement.  This  allows 
for  veridical  kinesthetic  and  proprioceptive  cues  to  augment  visual  information.  Screen  on  left  shows 
graphical  representation  of  electrotactile  stimulus  on  the  tongue  that  represents  closure  angle  (distance 
between  tactile  icons)  and  forces  (intensity  of  stimulus)  on  the  two  (right  and  left)  grasper  end  effectors. 

With  the  ShapeWraplll,  the  ACMI-SATS  system  can  accurately  track  arm,  torso  and  head 
position.  We  use  this  data  to  track  the  relative  position  and  orientations  of  the  wrists  to  drive  the 
position  and  orientation  of  the  Zeus  Microwrist  or  Mimic  EndoWrist®  and  operate  the  graspers 
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(opening  and  closing  the  hand  opens  and  closes  the  grasper  proportionally;  making  a  scissors 
cutting  motion  with  the  index  and  middle  finger  actuates  the  scissor  end  effector;  extending  the 
index  finger  alone  drives  blunt  dissection  with  a  closed  grasper).  This  allows  the  user  to  utilize 
kinesthetic  feedback  to  improve  SA.  We  also  added  head  tracking  (OptiTrack,  NaturalPoint,lnc., 
Corvalis,  OR)  to  the  control  input  stream  to  automatically  change  the  point  of  view  camera 
location  with  user  head  movement.  This  camera  control  mechanism  provides  motion  parallax 
tied  to  the  user's  x,  y  and  z  axis  head  movements  to  create  a  dynamic  viewpoint  that  supports 
intuitive  user  interaction.  As  the  user  moves  toward  and  away,  left  and  right,  or  up  and  down, 
with  respect  to  the  screen,  the  camera  system  zooms  in  and  out  or  shifts  the  view  in  the 
simulation  appropriately  to  create  an  effective  three  dimensional  (3D)  viewing  experience. 
Because  drastic  movements  would  affect  system  performance,  and  surgical  tasks  typically 
require  lower  dynamics,  simple  velocity  cut-offs  prevent  unintentional  motions  from  changing  the 
view  or  moving  the  robot.  This  allows  the  surgeon,  for  example,  to  rapidly  reposition  his  or  her 
hands  without  actively  clutching  while  also  preventing  a  sneeze  from  causing  a  robotic  mishap. 
While  this  model-based  approach  is  trivial  to  implement  with  simulators  (e.g.,  Yobotics  and 
Mimic  dV-Trainer),  we  can  accomplish  the  same  effect  with  actual  video  streams  by  integrating 
a  real-time  3D  model  creation  capability. 


Figure  10:  (top  row,  left  to  right)  Imaging  camera  system  (3  Kinect™  cameras)  mounted  to  endoscope 
positioner  robot  (Aesop  3000,  Computer  Motion,  Inc.,  Goleta,  CA)  as  seen  from  below;  Positioner  and 
camera's  retracted  (center)  and  extended  (right)  as  seen  from  above,  over  table  with  colored  objects, 
(middle  row)  3D  model  reconstructed  from  bottom  row  composite  camera  and  range  sensor  data  (bottom 
row)  showing  (left)  digital  zoom  and  rotation  of  image  and  model  (cameras  stationary);  (center)  model  and 
composite  image  with  Kinect™  array  positioner  retracted  and  (right)  with  positioner  extended. 
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We  initially  investigated  the  possibility  of  using  structured  light  (Tardif  &  Roy,  2005)  to  create 
models  through  the  endoscope,  however,  during  this  project,  the  Kinect™  camera  and  depth 
sensor  system  (Microsoft,  Corp.,  Redmond,  WA)  became  available.  This  sensor  provides,  at  low 
cost,  the  data  needed  to  generate  3-D  models  of  the  environment  under  study.  Using  three 
cameras  and  a  custom  simultaneous  localization  and  mapping  algorithm  (SLAM)  implemented 
in  AMI,  we  can  provide  parallax  cues  and  maintain  SA  regarding  the  location  of  objects  that  now 
fall  outside  the  endoscope  field  of  view  (Figure  10).  In  this  implementation,  the  camera  array 
moves  in  proportion  with  slow  head  movements  such  that  the  visual  scene  matches  live  camera 
images.  For  head  movements  that  have  velocities  or  trajectories  that  exceed  the  limitations  of 
the  camera  actuator,  the  visual  image  rotates  and  translates.  This  allows  for  the  imagery 
dynamics  to  match  the  user's  head  movements  without  lag  imposed  by  mechanical  endoscope 
movement  and  provides  depth  cues  for  3D  perception  without  requiring  the  surgeon  to  don 
special  goggles  or  sit  at  a  stereoscopic  display  console.  It  also  provides  a  point  of  view  that 
moves  in  an  intuitive  fashion  with  the  head,  such  that  moving  forward  or  backward  increases  or 
decreases  the  zoom  level,  respectively.  Vertical  and  horizontal  motion  causes  appropriate 
movement  of  the  point  of  view  in  the  simulation  and  the  parallax  to  provide  a  convincing  sense 
of  depth  (Figure  11).  This  provides  a  more  natural  perception  of  depth  than  the  current,  fixed 
stereoscopic  video  image  approach,  which  lacks  parallax.  The  camera  can  be  locked  (using  a 
hand  gesture)  in  a  given  viewpoint  or  moved  continuously  with  the  head  (which  provides  a 
visual  sensation  similar  to  that  of  an  open  surgical  procedure  and  removes  the  need  to  toggle 
between  endoscope  and  arm  controls).  Gestural  movements  of  the  fingers  allow  for  selection  of 
mode  (e.g.,  lock  position,  clutch,  activate  electrocautery,  etc.)  and  arm  (e.g.,  arm  two  or  three) 
without  using  foot-activated  switches. 


Figure  11:  Parallax  cues  emerge  in  the  3D  model  generated  from  the  composite  image  (center) 
generated  by  the  camera  array  (not  used  by  the  surgeon)  after  normalizing  for  angle.  Movement  of  the 
users  head  to  the  left  (left)  exposes  the  playing  card  (arrow)  and  covers  the  green  cone  partially.  Moving 
to  the  right  (right)  causes  the  nearer  blue  rectangle  to  obscure  the  card  and  fully  expose  the  green  cone. 

Multisensory  Implementations 

By  using  the  AMI  framework,  we  have  been  able  to  couple  a  wireless,  wearable  motion  capture 
system,  the  surgical  robot  environment  and  appropriate  sensory  feedback  displays  to  improve 
the  user  experience.  In  addition  to  the  visual  improvements  described  above,  we  have  fully 
implemented  a  set  of  tactile  and  auditory  cueing  capabilities  in  ACMI-SATS  to  allow  evaluation 
of  task  performance  with  research  participants  under  future  funding  vehicles.  A  connection 
between  the  AMI  system  and  the  Zeus  or  the  da  Vinci®  simulation  system  allowed  for  interactive 
simulation  and  testing  of  the  full  ACMI-SATS  concept.  Mimic  Technologies,  Inc.,  provided 
appropriate  software  hooks  and  documentation  to  enable  AMI  to  interface  to  simulation 
variables  including  setting  and  reading  the  states  of  the  endoscope  and  both  the  shafts  and 
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graspers  of  the  EndoWrists®.  In  addition  we 
can  retrieve  the  applied  forces,  (i.e.,  pushing 
and  pulling  forces)  on  tissues  as  well  as 
grasping  forces.  Figure  12  shows  a  flow  chart 
of  one  particular  ACMI-SATS  onfigura-tion 
detailed  in  the  following  sections. 

Kinesthetics :  Gestural  movements  of  the 
fingers  for  selection  of  mode  (e.g.,  lock 
position,  clutch,  activate  electrocautery,  etc.), 
arm  (e.g.,  arm  two  or  three)  and  head  allows 
the  user  to  simultaneously  change  viewpoint 
and  maneuver  the  robot  end  effectors.  The 
motion  parallax  provides  a  convincing  sense  of 
depth  in  the  2D  image.  With  the  imagery 
scaled  up  to  present  the  end  effectors  as 
approximately  the  same  size  as  the  users 
hands  (when  seated  or  standing  near  the 
center  of  the  room),  kinesthetic  feedback 
provides  additional  intuitive  cues.  This 
exemplar  tracks  hand  and  digit  position  data 
from  the  Measurand  AMI  software  agent,  with 
the  GestureGraspAgent  computing  the 
distance  between  the  fingers  of  interest 
(thumbs  and  indexes  of  each  hand). 


Measurand  system:  k 

Gloves 

Measurand  Agent 

Providing  outputs  from 
the  motion  capture 

system 

position  of  the  fingers 
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Control! 
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1  Image  of  the  pattern 

Basic  fir ainPart 

Agent 

Tiaiutorm  and  vend  the 
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Figure  12:  Flow  chart  of  a  full  ACMI-SATS  imple¬ 
mentation  for  the  Zeus  surgical  robot  (see  text). 


Tactile  Feedback:  The  GestureGraspAgent  sends  the  current  distance  between  the  fingers  and 
the  current  intensity  value  to  the  ZeusGripperBrainPortAgent.  The  agent  uses  the  values  to 
create  an  image  of  the  pattern  of  stimulation  sent  to  the  BrainPortAgent. 


Figure  13-:(top  row)  Three  different  symbologies  for  representing  grasper  closure  angle  on  the  tongue. 
(bottom  row)  Two  approaches  for  end  effector  electrotactile  representation,  (bottom  row  left)  One  open 
and  one  closed  grasper  (right  grasper  is  closed  firmly  resulting  in  higher  stimulus  intensity),  (bottom  row 
right)  An  alternate  representation  (Pattern  3  from  top  row)  of  the  tactile  symbology. 
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The  BrainPortAgent  includes  a  graphical  user  interface  (GUI)  that  provides  visual  feedback  to 
the  researcher  regarding  the  tactile  stimulus  (Figure  13).  The  BrainPortAgent  converts  the 
image  into  a  grayscale  intensity  map  delivered  to  the  BrainPort®  Intraoral  tongue  array  (Figure 
13  bottom  row).  The  tactile  iconography  includes:  1)  Soft-  the  intensity  first  increases  slowly 
when  the  grasper  encounters  an  object  and  then  more  rapidly  as  the  grasper  closes  (similar  to 
the  force  sensation  when  we  squeeze  a  soft  object);  2)  Medium-  the  intensity  remains  directly 
proportional  to  the  distance  between  the  fingers  (using  a  variable  gain  set  before  use);  3)  Hard- 
intensity  sharply  increases  as  soon  as  the  grasper  contacts  an  object  and  4)  Isointensity-  only 
grasper  closure  angle  is  represented,  with  intensity  held  constant. 


Raw  Data  Rec  Calibrate  VidcoTact  Raw  Data  R*<  Calibrate  Vidro  Tart  Ra»  Data  R»<  Calibrate  Vnt»oI  *it 


Figure  14:  (left)  Graphical  representation  of  VideoTact  display  of  the  robot  arm  positions  relative  to 
endoscope  orientation.  The  tactile  wedges  represent  the  projection  of  the  robot  arm  shafts  outside  the 
field  of  view  of  the  camera  back  to  the  entry  port  despite  movement  of  the  arms  or  camera  (center).  In 
addition,  increased  force  on  the  shafts  (e.g.,  from  contact  with  ribs)  causes  increased  intensity  of  the 
signal  (right).  This  allows  the  user  to  maintain  awareness  of  arm  positions,  movements  and  forces  that 
remain  outside  the  visual  display  of  the  camera  field  of  view  (black  rectangle  at  the  center). 

To  improve  perception  of  the  locations  of  the  robot  arms  beyond  the  field  of  view  of  the  surgeon, 
we  added  symbology  to  with  the  VideoTact  to  present  the  relative  positions  of  the  end  effector 
shafts  in  screen  coordinates  (Figure  14).  Stimulation  frequencies  of  1Hz,  4Hz  and  10Hz 
represented  each  arm  when  projected  onto  the  user’s  abdomen.  As  was  predicted,  10Hz  greatly 
facilitated  recognition.  While  initial  evaluation  indicated  that  three  moving  arms  were  more 
difficult  to  differentiate  effectively  than  just  one  or  two  arms,  further  symbology  development 
improved  the  saliency  of  all  three  robot  arm  icons. 


Figure  15:  Surround  Sound  (10.2  channel)  AMI  agent  GUI  for  spatially  positioned  audio  stimuli. 

Auditory  Feedback:  The  AMI  framework  also  allows  the  final  software  element  to  transform  any 
of  the  tactile  cues  into  spatial  auditory  cues  (Figure  15).  In  the  above  example,  the  agent 
manipulates  range  and  bearing  to  represent  the  grasper  closure  angle  and  force  in  surround 
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sound  using  fore/aft  sound  position  and  up/down  tone  frequency,  respectively.  The  agent  can 
play  multiple  sounds,  both  complex  and  simple  to  represent  data  (e.g.,  forces,  locations,  etc.)  in 
the  operative  field.  While  mapping  forces  onto  the  tongue  gave  the  user  a  finer  degree  of  control 
during  manipulation  of  the  objects  with  the  graspers,  the  audio  display  enabled  users  to  easily 
manipulate  two  objects  with  different  compliance  and  mass  simultaneously. 

The  agent  receives  data  from  the  Mimic  Simulation  Agent,  Collision  Manager  Agent,  or  other 
AMI  agents  and  applies  appropriate  attenuation  to  a  sound  signal  to  present  azimuth  and  range 
on  an  oval  array  of  10  speakers  and  two  subwoofers  (Avia  PowerStixx,  FuShan  Enterprises, 
Fremont,  CA)  mounted  in  the  ceiling  driven  by  a  12  channel  digital  audio  interface  (AudioFire12, 
Echo  Digital  Audio  Corp.,  Santa  Barbara,  CA).  The  dimensional  audio  also  proved  effective  for 
use  as  a  spatial  warning  system  in  a  collision  manager  feature  implemented  in  ACMI-SATS 
(described  below).  Though  designed  for  representation  on  the  TTI,  AMI  allows  for  evaluation  of 
collision  manager  functions  implemented  in  surround  sound. 
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Figure  16:  Multisensory  modality  capabilities  and  potential  uses  in  telesurgical  data  display. 


We  can  now  utilize  audio,  tactile  and  visual  cues  to  reduce  the  effort  required  to  complete  a 
simulated  surgical  task.  By  placing  feedback  related  to  forces  in  the  operative  field  on  tactile  and 
auditory  interfaces,  we  can  provide  veridical  sensory  information  without  the  haptic  force 
feedback  approach  that  interferes  or  competes  with  fine  motor  control.  The  flexibility  of  AMI 
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enables  efficient  design  of  research  protocols  that  will  seek  to  determine  the  best  set  of  multi- 
sensory  displays  and  symbology  strategies  for  specific  tele-surgery  applications  (Figure  16). 

Teamwork 

The  reduction  in  workload  associated  with  more  intuitive  multisensory  interfacing  should  allow 
increases  in  productivity,  such  that  a  single  surgeon  could  manage  more  than  one  case  at  a 
time,  in  concert  with  assistants  or  other  specialty  surgeons.  This  would  be  of  particular  use  in 
mass  casualty/battlefield  events  where  multiple  patients  in  multiple  facilities,  would  require 
interventions  from  multiple  surgical  specialties  who  may  be  physically  in  different  locations.  For 
example,  a  vascular  surgeon  could  repair  a  superior  mesenteric  arterial  injury  in  one  patient  and 
then  switch  immediately  to  a  previously  prepped  patient  in  need  of  a  right  renal  artery  repair  or 
ligation  (and  then  switch  back  quickly,  if  needed).  When  the  vascular  surgeon  releases  control 
of  the  second  robot,  the  urologist  takes  over  to  repair  a  severed  ureter  or  initiate  a  nephrectomy. 

Because  we  have  abstracted  the  control  interface  from  the  robot,  any  number  of  surgeons  could 
interface  individually  to  either  control  robots  located  in  different  facilities  or  interact  on  the  same 
case.  In  the  latter  case,  a  surgeon  and  a  first  assistant  could  share  control  of  the  multiple  robot 
actuators  on  the  same  case  to  improve  efficiency,  teaching  and  telementoring.  ACMI-SATS 
utilizes  gesture-based  switching  between  simultaneous  procedures  or  between  members  of  a 
surgical  team.  To  initiate  a  change,  the  surgeon  makes  a  clutch  gesture  (quickly  makes  a  fist 
with  both  hands)  to  lock  the  current  procedure  robot.  He  or  she  then  turns  to  face  the  video 
monitor  for  the  other  case  and  again  makes  the  clutch  gesture  to  release  its  robot  and  proceed 
with  the  case.  ACMI-SATS  uses  the  AMI  head-tracking  agent  to  determine  which  procedure  will 
receive  each  user's  commands.  It  also  switches  the  inputs  to  the  multisensory  displays  so  that 
the  surgeon  receives  tactile  and  auditory  feedback  referenced  to  the  currently  active  case  only. 
To  support  multiple  users  during  collaborative  robotic  procedures,  the  surgeon  can  release  one 
or  more  robot  arms  to  the  first  assistant  and  quickly  take  back  control  if  needed  (Figure  17). 


Figure  17:  ACMI-SATS  implementation  with  Mimic  dV-Trainer  showing  single  surgeon  management  of 
two  simultaneous  cases.  The  end  effectors  and  camera  view  move  with  the  user  in  one  case  at  a  time, 
selected  by  intuitive  user  gestures.  Tactile  interfaces  (lower  center)  provide  sensory  augmentation  for  the 
currently  active  case  by  providing  position  and  force  information  to  the  user. 
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Alternatively,  multiple  surgeons  can  operate  the  same  robot  (or  perhaps  two  separate  robots)  on 
the  same  patient  at  the  same  time  (Figure  18).  In  the  scenario  developed  by  IHMC,  an  expert 
specialty  surgeon  (e.g.,  cardiovascular)  mentors  a  fellow.  The  expert,  perhaps  located  in  his  or 
her  office,  takes  control  after  the  local  surgeon  preps  the  robot  and  patient.  The  expert  orients 
the  end  effectors  and  retractors,  reviews  the  procedure  with  the  fellow  or  first  assistant  and  then 
clutches  his  or  her  controls  (i.e.,  the  Mimic  dV-Trainer  control  console).  At  this  point  the  fellow 
can  unclutch,  assume  command  of  the  robot  and  begin  the  technical  portion  of  the  case.  To 
prevent  inadvertent  motion  of  the  robot  at  transfer,  the  trainee  must  orient  his  or  her  hands  to 
match  the  orientation  of  the  dV-Trainer  EndoWrists®  and  grasper  closure  angles  (as  visible  on 
the  video  screen)  before  ACMI-SATS  allows  the  end  effectors  to  move.  As  desired,  the  mentor 
can  unclutch,  match  wrist  orientation  and  grasper  angles,  and  take  over  control.  Either  can 
manipulate  the  third  robot  arm  at  any  times  in  order  to  adjust  retraction  or  to  cut  sutures.  This 
enables  effective  telementoring  and  interactive  telesurgical  training  and  collaboration. 


Figure  18:  Collaborative  control  of  a  single  procedure  with  ACMI-SATS.  (left)  The  surgeon  (plaid)  sets  up 
the  procedure  (using  the  mechanical  control  console  interface)  for  the  first  assistant  (red  shirt),  (right)  The 
surgeon  clutches  his  controls  and  the  first  assistant  performs  the  procedure  using  the  ACMI-SATS 
system.  The  surgeon  monitors  and  can  take  over  if  needed,  at  any  time.  The  two  collaborators  could 
control  the  robot  from  facilities  and  neither  is  required  to  be  collocated  with  the  patient. 

Gesture  based  control 

In  addition  to  direct  EndoWrist®,  grasper  and  camera  position/orientation  mapping  from  the 
ShapeWraplll  and  OptiTrak  motion  capture  systems,  ACMI-SATS  uses  specific  gestures  to 
manage  functions  normally  controlled  by  foot  pedals,  such  as  clutching  and  electrocautery,  as 
well  as  a  number  of  functions  not  available  on  a  surgical  robot  such  as  group  positioning,  "follow 
my  example"  and  "no-fly  zone"  definitions  for  collision  avoidance. 

Clutching:  Rather  than  rely  on  a  foot  pedal  actuation  to  clutch  and  unclutch  the  robot,  ACMI- 
SATS  uses  a  fist  clench  gesture  to  engage  the  clutch.  Clenching  the  right  hand  activates  the 
clutching  mechanism  for  the  arms  while  clenching  the  left  hand  clutches  the  camera.  Rotation  of 
a  clenched  fist  locks  that  part  of  the  robot  (akin  to  turning  a  deadbolt).  This  allows  the  surgeon 
to  lock  the  camera  movement,  for  example,  so  that  it  does  not  slave  to  his  or  her  head 
movement.  The  surgeon  then  continues  the  procedure  by  unclenching  both  fists.  Locking  the 
actuators  would  allow  a  surgeon  to,  for  example,  review  an  MRI  scan  on  a  radiology  display 
terminal  and  then  return  to  complete  the  procedure. 

No  Fly  Zone:  Surgical  mishaps  have  occurred  due  to  poor  surgeon  SA.  While  the  improved 
sensory  feedback  provided  by  ACMI-SATS  should  mitigate  loss  of  SA,  an  additional  gesture 
based  feature  allows  the  surgeon  to  designate  "no  fly  zones"  within  the  body  cavity  and  receive 
warnings  when  the  actuators  encroach  on  volumes  that  have  been  identified  by  the  surgeon  as 
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areas  to  avoid.  The  surgeon  positions  the  tip  of  robot  arm  1  and  defines  a  no  fly  zone  by 
touching  the  right  thumb  and  the  right  little  finger  after.  Opening  the  fingertips  places  a  default 
size  no-fly  zone  at  that  location  in  the  operative  field.  The  surgeon  can  define  larger  zones  with 
the  same  gesture  by  moving  arm  1  before  releasing  the  gesture.  This  defines  the  diameter  of  a 
spherical  volume  exclusion  zone.  Using  the  same  thumb/little  finger  gesture,  multiple  no  fly 
zones  can  be  defined  using  the  right  hand  and  deleted,  with  the  left  hand,  while  touching  near  a 
zone  with  robot  arm  2.  We  created  a  GUI  (figure  19)  to  ensure  accurate  monitoring  of  the 
process  during  testing,  however,  this  display  would  be  distracting  to  the  surgeon  and  the 
primary  use  is  to  maintain  SA  of  areas  outside  the  endoscope  field  of  view.  Therefore,  when  an 
instrument  approaches  or  enters  a  no  fly  zone,  the  AMI  agent  represents  the  information  either 
with  spatialized  audio  using  the  surround  sound  array  or  on  tactually  on  the  TTI.  AMI 
automatically  assigns  specific  sounds  or  vibration  frequencies  to  each  zone.  Should  any  part  of 
the  robot  end  effectors  start  approaching  too  closely,  the  assigned  sound  or  vibration  intensity 
increases  along  the  bearing  to  the  zone,  relative  to  the  endoscope  field  of  view. 


Figure  19:  No  fly  zone  defined  or  deleted  with  gestures  (inset)  appear  on  the  test  operator  GUI  for  test 
and  evaluation.  Escalating  spatially  relevant  tactile  or  audio  warnings  indicate  no  fly  zone  violations. 


"Follow  Mv  Example":  Repetitive  tasks  become  burdensome  in  robotic  surgery  in  comparison  to 
open  procedures.  In  open  procedures,  for  example,  surgeons  can  rapidly  place  a  line  of 
interrupted  sutures  with  even  spacing  and  tension,  but  they  must  slow  their  pace  during  robotic 
procedures  due  to  the  lack  of  qualitative  feedback  about  the  tissue  friability  and  suture  tension. 
ACMI-SATS  incorporates  a  "follow  me"  function  that  allows  the  surgeon  to  define  a  macro  of 
actions  and  then  instruct  the  robot  to  execute  a  series  of  these  actions,  adding  a  position  offset 
between  each  until  the  series  is  completed.  IHMC  demonstrated  this  function  in  the  "pick  and 
place"  practice  drill  available  in  the  Mimic  dV-Trainer  software  (Figure  20),  using  an  iterative 
task  similar  to  placing  surgical  staples  along  an  incision  line.  While  the  current  ACMI-SATS 
implementation  only  repeats  position  commands,  including  adding  offsets  between  iterations,  it 
could  be  extended  to  track  the  forces  on  the  graspers  and  actuator  arms.  With  force 
measurements  and  position  control,  the  software  agent  could  maintain  constant  force  or  indicate 
an  error  if  force  parameters  are  exceeded  (e.g.,  when  a  suture  needle  strikes  an  arterial  plaque 
during  an  anastomoses).  We  will  also  implement  gestural  control  methods  to  bypass  the  GUI 
when  defining  macros  in  future  research  activities. 
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Figure  20:  Follow  my  example  macro  for  repetitive  task  (i.e.,  align  randomly  placed  objects.  Step  1  (top), 
mark  position  of  all  jacks  (inset)  by  touching  the  grasper  to  each  one.  Step  2,  mark  goal  by  touching  a 
destination  and,  if  desired,  a  second  point  to  define  an  offset.  In  step  1  &  2,  the  agent  stores  the  X,  Y,  Z 
positions  of  the  jacks  and  the  goal  from  the  3D  position  of  the  grasper.  Step  three  (bottom)  a  previously 
defined  jack  grasping  macro  (middle)  is  nested  in  the  alignment  macro  and  the  system  iteratively  picks  up 
and  places  each  jack  (bottom,  left)  in  a  line  with  the  spacing  defined  in  step  2  (bottom,  right). 
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Group  positioning:  The  group  positioning  feature  allows  for  rapid,  gross  position  control  of  all 
three  arms  simultaneously.  This  feature  finds  use  when  moving  to  a  different  location  within  the 
operative  field  (as  might  happen  when  a  vascular  surgeon  hands  off  control  to  a  urological 
surgeon).  The  surgeon  first  clutches  both  the  camera  and  the  actuators  as  described  above.  He 
or  she  then  extends  the  index  finger,  middle  finger  and  thumb  on  the  right  hand  while  keeping 
the  left  hand  clenched  (Figure  21).  At  this  point  the  EndoWrists®  track  the  relative  position  and 
orientation  of  the  three  extended  digits  and  allow  the  surgeon  to  move  them  into  a  favorable 
starting  position.  Re-clutching  and  unclutching  both  hands  returns  to  normal  motion  tracking 
behavior  for  fine  position  control  and  continuation  of  the  procedure.  This  method  could  be  easily 
applied  to  control  robots  with  four  or  five  active  actuators  (in  addition  to  an  endoscope). 


Figure  21:  Group  positioning  gesture  with  left  hand  clenched  and  first  three  digits  of  the  right  hand 
extended  to  position  the  three  arms  simultaneously. 


Energy:  Rather  than  require  foot  pedals  for  activation  and  selection  of  energy  dissection  and 
cauterization,  ACMI-SATS  utilizes  a  clutch  followed  by  extending  index  and  middle  finger 
together  (thumb  and  other  two  fingers  clenched)  gesture  to  select  and  activate  energy 
dissection.  The  closed  grasper  moves  with  the  fingertip  position  to  dissect  electrosurgically. 
Cauterization  control  operates  in  a  similar  fashion,  however,  both  the  index  finger  and  thumb 
must  be  extended  to  select  electrocautery,  and  touching  the  fingertip  to  the  thumb  closes  the 
grasper  and  activates  the  energy. 

Relationship  to  award  Statement  of  Work 

We  developed  a  multisensory  interface  that  provides  visual,  audio  and  tactile  displays  as  well  as 
incorporates  veridical  kinesthetic  and  proprioceptive  cueing.  In  addition,  we  have  created  a 
generalizable,  intuitive  control  interface  for  surgical  robots  using  motion  capture  and  gesture 
detection  to  command  the  robotic  system.  Based  on  the  original  statement  of  work,  we  have 
completed  all  of  the  tasks  from  Specific  Aims  #  1  and  2  which  included: 


Specific  Aim  #  1:  Develop  a  multi-sensory  interface  to  augment  current  visual  and  haptic 

interfaces  that  will  include  of  spatial  audio  and  tactile  interfaces 

Task  1.  Modify  IHMC  Auditory  and  Tactile  interfaces  for  telesurgical  application 

la.  Select  tactile  displays  and  body  mounting  locations 

lb.  Define  intracorporeal  orienting  requirements 

lc.  Define  registration  method  for  marking  locations  of  organs  and  vessels 

ld.  Evaluate  tactile  and  audio  cueing  representations 
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1e.  Integrate  software  for  spatial  cueing 

Milestone  #1:  Spatial  audio  and,  tactile  interface  critical  design  review 
Task  2.  Integrate  IHMC  AMI  software  &  Mimic  dV-Trainer-Mantis  Duo 
2a.  Define  interface  over  Ethernet 

2b.  Test  transmission  of  dV-Trainer  state  variables  over  Ethernet  to  AMI 
2c.  Test  transmission  of  AMI  control  input  data  variables  to  dV-Trainer 
2d.  Develop  hardware  interface  to  existing  telesurgical  robotic  end  effectors 
2e.  Develop  software  interface  to  existing  telesurgical  robotic  end  effectors 
Milestone  #2:  Complete  software  Ethernet  interface  definition  and_  testing 
Specific  Aim  #2:  Develop  a  generic  telesurgical  robot  control  input  mechanism  that  allows  use 
of  normal  kinesthetic  hand  and  arm  movements  using  motion  capture  and  gesture  recognition 
Task  3.  Code  software  for  natural  movement  control  of  ACMI-SATS 
3a.  Define  and  integrate  motion  primitives  for  telesurgical  activities 
3b.  Bind  coded  motion  primitives  to  dV-Trainer  control  input  commands 
3c.  Code  user  interface 
3d.  Code  performance  evaluation  interface 

3e.  Test  and  evaluate  interface  function  with  tactile  and  audio  displays 
3f.  Code  finite  state  machine  (FSM)  algorithm  for  gesture  recognition 
3g.  Test  and  evaluate  gesture  based  commands  in  dV-trainer  simulation 
Milestone  #3:  ACMI-SATS  system  design  complete 
Task  4.  Demonstration  and  evaluation 

4a.  Prototype  system  functional  verification  and  testing 
4b.  Demonstration  and  evaluation  with  subject  matter  expert  Co-Investigator 
4c.  Analyze  results  for  presentation  at  the  Society  of  American  Gastrointestinal  and 
Endoscopic  Surgeons  (SAGES)  annual  conference 

Milestone  #4:  Demonstrate  ACMI-SATS  system  with  simultaneous  simulated  procedures 
Milestone  #5:  Draft  manuscript  for_submission  to  Surgical  Endoscopy  (journal} 

Key  research  accomplishments 

•  We  successfully  integrated  our  AMI  software  architecture  with  a  physical  four  arm  surgical 
robot  (Zeus)  and  with  a  commercially  available  tele-surgical  simulation  (dV-Trainer) 

•  We  created  a  physics-based  kinematically  accurate  graphical  simulation  of  the  Zeus  robot 

•  We  integrated  spatial  audio,  tactile  and  large  format  visual  displays  for  improved  surgeon 
situation  awareness. 

•  We  have  evaluated  a  number  of  electrotactile  and  auditory  symbologies  for  representation 
of  the  positions  and  forces  on  the  end  effectors  as  well  as  relative  locations  and  forces  on 
the  robot  arm  shafts. 

•  We  have  integrated  a  wireless  full-body,  wearable  motion  capture  system  (ShapeWraplll) 
and  head  tracking  (OptiTrack)  to  allow  full  freedom  of  movement  for  the  surgeon. 

•  We  implemented  gesture  based  control  elements  into  the  ACMI-SATS  testbed  to  provide 
intuitive  mode  switching  and  tele-robotic  system  control 

•  We  implemented  a  1:1  scaling  between  arm  and  hand  movements  and  the  visual 
representation  of  the  endoscopic  view  to  ensure  useful  kinesthetic,  proprioceptive  and 
motion  parallax  cues. 

Reportable  outcomes 

We  presented  our  preliminary  approach  and  results  at  the  Aerospace  Medical  Association’s  81st 
Annual  Scientific  meeting  two  posters  (general  section  and  Emerging  Technologies  section) 
describing  the  ACMI-SATS  system  to  the  Society  of  American  Gastrointestinal  and  Endoscopic 
Surgeons  annual  scientific  session  (SAGES  11). 
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A  journal  paper  has  been  drafted  and  will  be  submitted  to  Surgical  Endoscopy. 

Conclusion 

The  ACMI-SATS  project  developed  a  testbed  that  allows  for  testing  and  evaluation  of 
hypotheses  related  to  tele-surgical  robotic  human-centered  surgeon  interfaces,  cognitive 
workload,  efficiency  and  situation  awareness.  The  ACMI-SATS  testbed  includes  both  physical 
surgical  robotic  capability  (Zeus)  and  simulation  environments  (Zeus/Yobotics,  da  Vinci®/Mimic) 
to  maximize  the  efficiency  of  system  development.  We  have  completed  the  testbed  hardware 
and  software,  and  our  research  staff  has  completed  functional  verification  of  ACMI-SATS 
augmentations.  Force  feedback  systems  that  backdrive  against  the  user's  hands  can  cause  the 
surgeon  to  increase  force  against  the  hand  controllers  and  lead  to  rapid  velocity  changes  when 
the  position  controlled  actuators  interact  with  tissues  with  variable  compliance.  ACMI-SATS 
avoids  this  issue  by  providing  veridical  sensory  feedback  without  interfering  with  fine  motor 
control  movements  of  the  hands.  The  completed  ACMI-SATS  testbed  will  support  future  studies 
to  develop  optimal  tele-surgical,  tele-mentoring  and  remote  operations  technologies  and 
methodologies  for  both  military  and  civilian  applications. 

Future  work  would  include  extending  the  SLAM  3D  models  to  maintain  a  wider  field  of  view  in 
the  model  as  the  camera  is  moved,  reducing  the  size  of  the  depth  sensor  (e.g.,  Aerius 
Photonics,  LLC,  makes  a  number  of  small  rangefinders  that  could  be  integrated  into 
endoscopes),  and  testing  performance  gains  in  standard  surgical  procedures  (as  well  as  those 
enabled  only  by  the  capabilities  of  ACMI-SATS).  Because  the  ShapeWraplll  system  used  in  this 
prototype  was  designed  primarily  for  capturing  movement  for  animation  and  gaming,  it  is  more 
obtrusive  than  desirable  for  an  operational  ACMI-SATS  system.  More  compact  motion  sensing 
systems  could  eventually  remove  the  requirement  for  body  worn  technologies  altogether.  While 
this  team  and  others  (Dunn  2011)  developed  solutions  for  sensing  and  modeling  the  operative 
field  using  commercial  off  the  shelf  (COTS)  sensors  such  as  the  Microsoft  Kinect™,  we 
determined  that  this  device  is  not  adequate  for  surgeon  motion  capture.  It  lacks  the  resolution  to 
track  the  orientations  and  positions  of  the  individual  fingers,  and  the  relatively  low  sample  rate 
prevents  effective  real-time  signal  processing,  which  would  be  needed  for  repeatability  and 
accurate  registration.  Future  robotic  end  effectors  that  replace  the  position  controlled 
mechanical  drives  in  the  actuators  with  force  controlled  drives  would  allow  for  better  control  of 
the  end  effectors  when  working  with  tissues  with  variable  compliance.  This  would  enable  a  more 
functional  force  following  capability  for  the  "follow  my  example"  function  of  ACMI-SATS  and 
prevent  mishaps,  such  as  inadvertent  tissue  injury,  that  manifest  with  position  control. 

The  military  clearly  would  benefit  from  improved  telemedical  systems  that  allow  more  rapid 
delivery  of  definitive  medical  care  to  wounded  warfighters.  ACMI-SATS  could  complement  DoD 
projects  such  as  the  DARPA  Trauma  Pod  that  will  reduce  the  time  between  battlefield  injury  and 
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definitive  treatment  by  providing  a  method  for  a  surgeon  geographically  removed  from  the 
soldier  to  rapidly  intervene,  without  having  to  wait  for  transportation  to  a  field  hospital, 
stabilization  and  airlift  to  a  level  I  trauma  center.  The  civilian  world  of  medicine  could  also  benefit 
tremendously  from  telesurgical  systems  enabled  by  ACMI-SATS.  The  ability  to  perform  surgery 
from  a  distance  will  prove  extremely  useful  in  places  and  situations  where  distance,  disaster, 
lack  of  infrastructure,  or  the  prohibitions  of  cost  may  preclude  the  presence  of  expert  surgeons. 
Removing  the  need  for  a  medical  expert’s  actual  presence  in  these  situations  creates 
opportunities  to  provide  essential  medical  care  to  persons  who  might  otherwise  not  receive  it. 

References 

Bach-y-Rita,  P.,  Collins,  C.  C.,  Saunders,  F.,  White,  B.,  &  Scadden,  L.  (1969).  Vision 
substitution  by  tactile  image  projection.  Nature,  221,  963-964. 

Bach-y-Rita,  P.  (1972).  Brain  mechanisms  in  sensory  substitution.  New  York:  Academic  Press. 
Bach-y-Rita,  P.  (1995).  Nonsynaptic  diffusion  neurotransmission  and  late  brain  reorganization. 
New  York:  Demos-Vermande. 

Bach-y-Rita,  P.,  Kaczmarek,  K.  A.,  Tyler,  M.  E.,  &  Garcia-Lara,  J.  (1998).  Form  perception  with 
a  49-point  electrotactile  stimulus  array  on  the  tongue:  A  technical  note.  Journal  of 
Rehabilitation  Research  and  Development,  35(4),  427-430. 

Bar-Cohen,  Y.,  Mavroidis,  C.,  Bouzit,  M.,  Dolgin,  B.,  Harm,  D.L.,  Kopchok,  G.E.,  &  White,  R. 
(2001,  March).  Virtual  reality  robotic  tele-surgery  simulations  using  MEMICA  haptic  system. 
Proceedings  of  SPIE's  8th  Annual  International  Symposium  on  Smart  Structures  and 
Materials,  Newport,  CA.  Paper  No.  4329-47 
Bethea,  B.T.,  Okamura,  A.M.,  Kitagawa,  M.,  Fitton,  T.P.,  Cattaneo,  S.M.,  Gott,  V.L., 

Baumgartner,  W.A.,  &  Yuh,  D.D.  (2004,  June).  Application  of  Haptic  Feedback  to  Robotic 
Surgery.Journa/  of  Laparoendoscopic  &  Advanced  Surgical  Techniques.  74(3):  191-195. 
Blankenburg,  F.,  Ruff,  C.  C.,  Deichmann,  R.,  Rees,  G.,  Driver,  J.  (2006).  The  Cutaneous  Rabbit 
Illusion  Affects  Human  Primary  Sensory  Cortex  Somatotopically.  PLoS  Biology  4(3),  e69. 
Bremer,  C.  D.,  Pittenger,  J.  B.,  Warren,  R.,  Jenkins,  J.  J.,  (1977).  An  illusion  of  auditory  saltation 
similar  to  the  cutaneous  “rabbit”.  American  Journal  of  Psychology  90(4),  645-654. 

Briggs,  S.  J.  &  Srinivisan,  M.  A.  (2001).  Haptic  interfaces.  In  K.  Stanney  (Ed.)  Handbook  of 
Virtual  Environments:  Design,  Implementation,  and  Applications  (pp.  93-116).  Mahwah,  NJ: 
Erlbaum. 

Broderick,  T.  J.  (2006).  High  Altitude  Platforms  Mobile  Robotic  Tele-surgery  (HAPsMRT). 

Annual  Report  Contract  Number  W81XWH0520080.  Report  No.  A140174. 

Buonomano,  D.  V.,  Merzenich,  M.  M.  (1998).  Cortical  plasticity:  from  synapses  to  maps.  Annual 
Review  of  Neuroscience  27,149-186 

Burnett,  M.  (2007,  Jan).  Robotic  relief.  Military  Medical/NBC  Technology,  77(1). 

Calhoun,  G.  L.,  Draper,  M.  H.,  Ruff,  H.  A.,  &  Fontejon,  J.  V.  (2002).  Utility  of  a  tactile  display  for 
cueing  faults.  Proceedings  of  the  Human  Factors  and  Ergonomics  Society  46th  Annual 
Meeting,  2144-2148. 

Cholewiak,  R.  W.  (1976).  Satiation  in  cutaneous  saltation.  Sensory  Processes,  1,  163-175. 

Cohn,  J.,  Burns,  J.,  Helmick,  J.,  &  Meyers,  C.  (2000).  Training-transfer  guidelines  for  virtual 
environments.  Proceedings  of  the  22nd  Interservice/Industry  Training,  Simulation  and 
Education  Conference.  Orlando  FL. 

Craig,  J.  C.  (1968).  Vibrotactile  spatial  summation.  Perception  &  Psychophysics,  4,  351-354. 
Culjat,  M.O.,  King,  C.H.,  Franco,  M.L.,  Bisley,  J.W.,  Dutson,  E.,  &  Grundfest  W.S.  (2008). 
Pneumatic  Balloon  Actuators  for  Tactile  Feedback  in  Robotic  Surgery.  Industrial  Robot, 

35(5),  449-455. 

Diamond,  D.  D.,  Kass,  S.  J.,  Andrasik,  F.,  Raj,  A.  K.,  &  Rupert,  A.  H.  (2003).  Vibrotactile  cueing 
as  a  master  caution  system  for  visual  monitoring.  Human  Factors  and  Aerospace  Safety,  2, 
339-354. 


Approved  for  public  release;  distribution  unlimited  25 


Dunn,  R.  (201 1 ).  UW  students  adapt  gaming  hardware  for  surgery.  The  Daily  of  the  University 
of  Washington.  132{  1 1 ):  1 

Eriksson,  L.,  Berglund,  A.,  Willen,  B.,  Svensson,  J.,  Petterstedt,  M.,  Carlander,  O.,  Lindahl,  B.,  & 
Allerbo,  G.  (2008,  Sept).  On  visual,  vibrotactile,  and  3D  audio  directional  cues  for  dismounted 
soldier  waypoint  navigation.  Proceedings  of  the  Human  Factors  and  Ergonomics  Society, 
New  York. 

Finkel,  L.  H.  (1990).  A  model  of  receptive  field  plasticity  and  topographic  map  reorganization  in 
the  somatosensory  cortex.  In  S.  J.  Hanson  &  C.  R.  Olsen,  (Eds.),  Connectionist  Modeling 
and  Brain  Function:  The  Developing  Interface  (pp.  164-192).  Cambridge:  MIT  Press. 

Franke,  J.,  Zaychik,  V.,  Spura,  T.  &  Alves,  E.  (2005).  Inverting  the  Operator/Vehicle  Ratio: 
Approaches  to  Next  Generation  UAV  Command  and  Control,"  In  Proceedings  of  Association 
for  Unmanned  Vehicle  Systems  International  and  Flight  International,  Unmanned  Systems 
North  America.  Baltimore,  MD. 

Fuchs,  S.,  Johnston,  M.,  Hale,  K.S.,  &  Axelsson,  P.  (2008,  Sept).  Results  from  pilot  testing  a 
system  for  tactile  reception  of  advanced  patterns  (STRAP).  Proceedings  of  the  Human 
Factors  and  Ergonomics  Society,  New  York. 

Gallagher,  A.  G.,  Ritter,  E.  M.,  Champion,  H.,  Higgins,  G.,  Fried,  M.  P.,  Moses,  G.,  Smith,  C.  D., 
&  Satava,  R.  M.  (2005).  Virtual  reality  simulation  for  the  operating  room:  Proficiency-based 
training  as  a  paradigm  shift  in  surgical  skills  training.  Annals  of  Surgery,  241 ( 2),  364-372. 

Geldard,  F.  A.  (1975).  Visual  Saltation  -  Cutaneous  Rabbit  Invades  Visual-Field.  Bulletin  of  the 
Psychonomic  Society.  6(NB4):  433-433. 

Geldard,  F.  A.  &  Sherrick,  C.  E.  (1972).  The  cutaneous  "rabbit":  a  perceptual  illusion.  Science 
178(51),  178-179. 

Gilliland,  K.,  Schlegel,  R.,  &  Shehab,  R.  (2005).  Remote  manipulator  system  operator  training 
and  performance  metrics.  In  Bioastronautics  investigators  workshop.  January  10-12, 
Galveston,  TX. 

Hagen,  M.  E.,  Meehan,  J.  J.,  Inan,  I.,  &  Morel,  P.  (2008).  Visual  clues  act  as  a  substitute  for 
haptic  feedback  in  robotic  surgery.  Surgery  and  Endoscopy,  22,  1505  -  1508. 

Hannaford,  B.,  &  Venema,  S.  (1995).  Kinesthetic  displays  for  remote  and  virtual  environments. 

In  W.  Barfield  and  T.  Furness  (Eds.)  Virtual  Environments  and  Advanced  Interface  Design 
(pp.  415-436).  Oxford  University  Press:  Oxford. 

He,  F.,  &  Agah,  A.  (2001).  Multi-modal  human  interactions  with  an  intelligent  interface  utilizing 
images,  sounds,  and  force  feedback.  Journal  of  Intelligent  &  Robotic  Systems,  32(2),  171- 
190. 

Hirshauer,  S.  (1991).  The  manufacture  of  bodies  in  surgery.  Social  Studies  of  Science,  21,  279- 
319. 

Kitagawa,  M.,  Dokko,  D.,  Okamura,  A.M.,  &  Yuh,  D.D.  (2005). Effect  of  sensory  substitution  on 
suture-manipulation  forces  for  robotic  surgical  systems.  Journal  of  Thoracic  and 
Cardiovascular  Surgery,  129,  151-158. 

Kupers,  R.,  Sampaio,  E.,  Moesgaard,  S.,  Gjedde,  A.,  &  Ptito,  M.  (2003,  June).Acf/Vaf/'on  of 
visual  cortex  by  electrotactile  stimulation  of  the  tongue  in  early-blind  subjects.  Ninth  Annual 
Meeting  of  the  Organization  for  Human  Brain  Mapping,  New  York,  NY. 

McGrath,  B.  J.,  Estrada,  A.,  Braithwaite,  M.  G.,  Raj,  A.  K.,  &  Rupert,  A.  H.  (2004).  Tactile 
Situation  Awareness  System  Flight  Demonstration  Final  Report  (US  Army  Aeromedical 
Research  Laboratory  Report  No.  USAARL  2004-10). 

Merlo,  J.,  Gilson,  R.,  &  Hancock,  P.A.  (2008,  Sept).  Crossmodal  congruency  benefits  for  tactile 
and  visual  signaling.  Proceedings  of  the  Human  Factors  and  Ergonomics  Society,  New  York. 

Merzenich,  M.  M.,  Recanzone,  G.,  Jenkins,  W.  M.,  Allard,  T.  T.,  &  Nudo,  R.  J.  (1988).  Cortical 
representational  plasticity.  In  P.  Rakic  &  W.  Singer  (Eds.),  Neurobiology  of  neocortex  (pp.  41- 
67).  New  York:  John  Wiley  &  Sons  Limited. 


Approved  for  public  release;  distribution  unlimited  26 


Morimoto,  A.K.,  Foral,  R.D.,  Kuhlman,  J.L.,  Zucker,  K.A.,  Curet,  M.J.,  Bocklage,  T.,  MacFarlane, 
T.I.,  &  Kory,  L.  (1997).  Force  sensor  for  laparoscopic  Babcock.  Studies  in  Health  Technology 
lnformatics.39,  354-61. 

New  Scientist  (1972).  "Technology  Review:  Quick  on  the  draw  in  all  directions."  IPC  Magazines, 
Ltd,  London,  pp.  513. 

Olson,  J.  M.,  Raj,  A.  K.,  Thomas,  R.  &  Davis,  J.  (2004,  May).  Tactile  display  landing  safety, 
situational  awareness,  and  workload  reduction  improvements  for  the  space  shuttle. 
Proceedings  of  the  Aerospace  Medical  Association  75th  Annual  Scientific  Meeting, 

Anchorage  AK. 

Pueschel,  M.  (2006).  Demand  rises  forLSTAT platforms.  From  US  Medicine  Website. 
http://www.usmedicine.com/article.cfm?articlelD=614&issuelD=84 

Raj,  A.  K.,  Kass,  S.  J.,  &  Perry,  J.  F.  (2000).  Vibrotactile  displays  for  improving  spatial 

awareness.  Proceedings  of  the  Human  Factors  and  Ergonomics  Society  Annual  Meeting  (pp 
1:181-184).  San  Diego,  CA. 

Rosen,  J.,  &  Hannaford,  B.  (2006).  Doc  at  a  Distance:  Robot  Surgeons  Promise  to  Save  Lives 
in  Remote  Communities,  War  Zones,  and  Disaster-stricken  Areas.  IEEE  Spectrum,  43(10), 
34-39. 

Sampaio,  E.,  Maris,  S.,  &  Bach-y-Rita,  P.  (2001).  Brain  plasticity:  ‘visual’  acuity  of  blind  persons 
via  the  tongue.  Brain  Research,  908,  204-207. 

Sarter,  N.  B.  (2001).  Multimodal  communication  in  support  of  coordinative  functions  in  human- 
machine  teams.  Journal  of  Human  Performance  in  Extreme  Environments,  5,  50-54. 

Satava,  R.  (1995).  Virtual  reality,  tele-surgery,  and  the  new  world  order  of  medicine.  Journal  of 
Image  Guided  Surgery,  1, 12-16. 

Satava,  R.  (2005a).  Tele-surgery,  robotics,  and  the  future  of  telemedicine.  European  Surgery, 
37(5),  304  -  307. 

Satava,  R.  (2005b).  New  horizons  for  pre-hospital  and  battlefield  casualty  care.  From  US 
Medicine  Website.  http://www.usmedicine.com/column.cfm?columnlD=210&issuelD=79 

Tardif,  J.-P.,  Roy  S.,  A  MRF  formulation  for  coded  structured  light.  The  5th  International 
Conference  on  3-D  Digital  Imaging  and  Modeling  (3DIM’2005). pp  22-29. 

Tavakoli,  M.,  Patel,  R.V.,  &  Moallem,  M.  (2005).  "Haptic  interaction  in  robot-assisted  endoscopic 
surgery:  A  sensorized  end  effector,"  The  International  Journal  of  Medical  Robotics  and 
Computer  Assisted  Surgery.  1(2),  53-63. 

Tyler,  M.  E.,  Danilov,  Y.  P.,  Bach-y-Rita,  P.  (2003).  Closing  an  open-loop  control  system: 
Vestibular  substitution  through  the  tongue.  Inti.  J.  of  Integrative  Neuroscience,  2(2),  159-166. 

Vassiliades,  T.A.  (2006).  Robotics  in  cardiac  surgery.  International  Journal  of  Medical  Robots, 

2(1),  4 -6. 

Wagner,  C.R.,  Stylopoulos,  N.,  Jackson,  P.G.,  &  Howe,  R.D.  (2007).  The  Benefit  of  Force 
Feedback  in  Surgery:  Examination  of  Blunt  Dissection.  Presence:  Teleoperators  &  Virtual 
Environments  16(3),  252-262. 

Walcott,  E.  C.,  &  Langdon,  R.  B.  (2001 ).  Short-term  plasticity  of  extrinsic  excitatory  Whitsel,  B. 

L.,  Favorov,  O.  V.,  Tommerdahl,  M.,  Diamond,  M.  E.,  Juliano,  S.  L.,  &  Kelly,  D.  G.  (1989). 
Dynamic  processes  governing  the  somatosensory  cortical  response  to  natural  stimulation.  In 
J.  S.  Lund  (Ed.),  Sensory  processing  in  the  mammalian  brain  (pp.  84-116).  New  York:  Oxford 
University  Press. 

Whitsel,  B.  L.,  Favorov,  O.  V.,  Tommerdahl,  M.,  Diamond,  M.  E.,  Juliano,  S.  L.,  &  Kelly,  D.  G. 
(1989).  Dynamic  processes  governing  the  somatosensory  cortical  response  to  natural 
stimulation.  In  J.  S.  Lund  (Ed.),  Sensory  processing  in  the  mammalian  brain  (pp.  84-116). 

New  York:  Oxford  University  Press. 

Zhou,  M.,  Tse,  S.,  Dervianko,  A.,  Jones,  D.B.,  Schwaitzberg,  S.D.,  &  Cao,  C.G.L.  (2008,  Sept). 
The  Effect  of  Haptic  Feedback  on  Laparoscopic  Suturing  and  Knot-tying:  A  Learning  Curve 
Study.  Proceedings  of  the  Human  Factors  and  Ergonomics  Society,  New  York. 


Approved  for  public  release;  distribution  unlimited  27 


Appendices 

Appendix  1 :  Abstracts 

Raj,  A.  K.  &  Cameron,  J.  D.  (2010).Multisensory  Augmentation  of  Telemedicine 
Systems, Aerospace  Medical  Association  81st  Annual  Scientific  Meeting,  Phoenix,  AZ. 

Introduction:  Modern  electronics,  software  and  networking  technologies  have  enabled  the 
development  of  telemedicine  capabilities  that  bring  medical  expertise  to  remote  locations. 
Additionally,  telemedicine  enables  telementoring  to  allow  more  individuals  to  benefit  from 
interactions  with  expert  practitioners.  The  aerospace  medical  community  has  embraced  these 
novel  concepts,  which  have  the  potential  to  dramatically  improve  delivery  of  care  to  deployed 
units  or  vehicles  in  flight.  The  available  interfaces,  however,  provide  primarily  visual  information 
with  some  additional  verbal  auditory  data.  This  prevents  the  physician  from  experiencing  the 
rich  multisensory  cues  utilized  in  normal  one-on-one  patient  or  training  interactions.  Previously 
we  have  developed  an  anthro-centric  multisensory  interface  (ACMI)  system  that  can  manage 
multiple  types  of  data  streams  and  display  information  to  users  via  visual,  audio  and  tactile. 

Methods:  The  adaptive  multiagent  integration  (AMI)  software  agent  architecture  was  used  to 
integrate  video  imagery,  surround  sound  audio  and  tactile  representations  of  information  during 
telemedical  simulations.  This  ACMI  implementation  provides  typical  video  feeds  with  adjustable 
levels  of  visual  graphic  overlay.  Multiple  microphones  capture  spatial  relationships  of  the 
auditory  environment  (represented  using  surround  sound).  Combined  with  tactile  spatial 
orientations  interfaces  that  provide  perception  of  items  outside  the  visual  field  provided  by  the 
camera,  these  displays  improve  user  situation  awareness  (SA). 

Results:  Performance  (time  to  complete,  number  of  errors)  improved  for  individuals  when  using 
telemedicine  apparatus  with  ACMI  vs.  no  ACMI  enabled  for  a  series  of  telemedicine  relevant 
simulated  tasks. 

Discussion:  Telemedicine  holds  great  promise  for  use  in  aerospace  environments. 
Augmentation  with  multisensory  displays  could  enhance  effectively  and  expand  capabilities  and 
applications  without  increasing  user  workload. 

Learning  Objective  1 

Current  methods  for  multisensory  interface  implementations  will  be  described. 

Learning  Objective  2 

Techniques  for  improving  multisensory  situation  awareness  without  increasing  system 
bandwidth  usage  will  be  discussed. 

Learning  Objective  3 

Effectiveness  of  multisensory  interface  in  telemedicine  applications  will  be  discussed. 
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Objective  of  Technique 

While  telerobotic  surgical  systems  augment 
minimally  invasive  surgery  techniques  and 
reduce  surgical  trauma,  they  increase  surgeon 
workload  by  limiting  sensory  feedback.  Unlike 
open  or  laparoscopic  procedures,  surgical  robots 
isolate  the  surgeon  from  tactile,  proprioceptive, 
kinesthetic  and  orientation  cues,  which  provide 
non-visual  inputs  that  help  maintain  situation 
awareness  (SA).  The  robot  control  system, 
however,  employs  sensors  for  torque,  position, 
velocity  and/or  strain  sensors  to  maintain 
accurate  dosed-loop  servo  motion,  which  could 
help  inform  the  surgeon's  SA.  Though  sitting  at 
a  console  compares  favorably  to  standing 
astride  an  operating  table  using  laparoscopic 
instruments,  removing  all  restrictions  on 
surgeon  motion  and  posture  would  allow 
telerobotic  surgeons  to  move  freely  and  reduce 
both  physical  and  cognitive  fatigue,  maintain 
awareness  of  such  changes.  Lastly,  current 
surgical  robot  interfaces  provide  little,  if  any, 
feedback  to  help  the  surgeon  maintain  SA  when 
changing  from  primary  instruments  to 
secondary  instruments  (e.g.  third  effector, 
endoscope,  etc.). 
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Figure  1:  ACMI-SATS  surround  sound  processing  can 
arbitrarily  piece  the  relative  location  (range  and  azimuth)  of  a 
system  data  point. 

The  Anthro-Centric  Multisensory  Interface  for 
Sensory  Augmentation  of  Tele-Surgery  (ACMI- 
SATS)  project  seeks  to  empower  telerobotic 
surgery  with  many  of  the  sensory  and 
kinesthetic  cues  available  in  open  surgical 
procedures.  This  improves  the  effectiveness  of 
tele-surgery  by  allowing  a  surgeon  to  utilize 
tactile,  spatial  audio  and  three-dimensional 
visual  cues  as  well  as  meaningful  proprioceptive 
and  kinesthetic  information. 


Description  of  the  Methods 

The  Florida  Institute  for  Human  and  Machine 
Cognition  (IHMC)  developed  the  ACMI-SATS 
system  with  both  simulated  and  actual  surgical 
robot  systems.  It  provides  an  integrated 
architecture  with  a  wide  field  of  view,  pseudo- 
three  dimensional  visual  operative  field, 
surround  sound  for  spatially  relevant  audio 
when  selecting  instruments  and  multiple  tactile 
interfaces  that  represent  instrument  dynamics. 
ACMI-SATS  provides  a  natural,  free  motion 
control  interface  that  uses  a  wearable,  wireless 
motion-capture  system  to  track  head,  torso, 
arm,  wrist  and  finger  movements.  Natural 
motions  drive  the  movement  of  the  robotic 
instruments  and  the  endoscope.  The  surgeon 
can  stand  or  sit  without  any  external  restrictions 
and  proprioceptive  and  kinesthetic  sensations 
map  to  the  visual  presentation  at  a  1:1  scale. 
Motion  capture  enables  gestural  control  to 
manage  mode  changes  (such  as  clutching  or 
switching  control  to  a  different  arm). 

Preliminary  Results 

Initial  evaluations  on  the  ACMI-SATS  test  bed 
indicate  that  novices  (non-surgeons)  can  learn 
to  control  and  manipulate  the  robotic  end 
effectors  to  perform  minimally  invasive  surgical 
training  tasks  in  simulation  quickly  and  with  low 
cognitive  effort. 


Figure  2  ACM!  SATS  tactile  tongue  symbology  for  grasper 


closure  angle  and  force. 
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Figure  3:  ACMI-SATS  in  software  robotic  surgical  simulation 
(dV-trainer,  mimic  Technologies,  Inc.,  Seattle,  WA)  demon¬ 
strating  head  and  full  body  motion  capture  integrated  noth 
auditory  and  tactile  information  displays  (inset  shows 
graphical  representation  of  manipulator  shafts  and  grasper 
angle;  each  also  indicates  the  forces  on  the  end  effectors). 

The  sensory  interfaces  augment  understanding 
by  providing  additional  information  to  the 
surgeon  Intuitively,  without  overloading  visual  or 
auditory  sensory  channels.  The  motion-capture 
and  gestural  control  interface  allows  the 
surgeon  to  use  more  natural,  unrestricted 
movements  to  control  robot  actions  and  mode 
changes.  By  mapping  the  visual  scale  directly  to 
motion  of  the  arms,  the  surgeon  can  maintain 
awareness  of  instrument  positions  even  when 
they  are  no  longer  with  the  endoscope  field  of 
view.  Any  surgical  robot  platform  could  Integrate 
ACMI-SATS  and  allow  surgeons  to  learn  as  well 
as  perform  more  procedures  in  a  given  time 
frame,  with  less  effort  and  with  fewer  surgical 
errors. 


Figure  4  ACMI-SATS  user  performing  task  training  using 
natural  arm  and  hand  movements  and  gestures,  while  main¬ 
taining  SA  of  forces  on  and  locations  of  robotic  manipulators 
and  anatomical  structures  through  visual,  audio  and  tactile 
feedback. 
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Objective  of  Technique 

While  telerobotic  surgical  systems  effectively 
augment  minimally  invasive  surgery  techniques 
and  reduce  surgical  trauma,  they  increase 
surgeon  workload  by  limiting  sensory  feedback. 
Unlike  open  or  laparoscopic  procedures,  surgical 
robots  isolate  the  surgeon  from  tactile, 
proprioceptive,  kinesthetic  and  orientation  cues, 
which  provide  non-visual  inputs  that  help 
maintain  situation  awareness  (SA).  The  robot 
control  system,  however,  employs  sensors  for 
torque,  position,  velocity  and/or  strain  sensors 
to  maintain  accurate  closed-loop  servo  motion, 
which  could  help  inform  the  surgeon's  SA. 
Though  sitting  at  a  console  compares  favorably 
to  standing  astride  an  operating  table  using 
laparoscopic  instruments,  removing  all 
restrictions  on  surgeon  motion  and  posture 
would  allow  telerobotic  surgeons  to  move  freely 
and  reduce  both  physical  and  cognitive  fatigue, 
maintain  awareness  of  such  changes.  Lastly, 
current  surgical  robot  interfaces  provide  little,  if 
any,  feedback  to  help  the  surgeon  maintain  SA 
when  changing  from  primary  instruments  to 
secondary  instruments  (e.g.  third  effector, 
endoscope,  etc.). 


Figure  1  ACMI-SATS  tactile  tongue  symbology 
closure  angle  and  force,  (inset)  ACMI-SATS  surround  sound 
processing  can  arbitrarily  place  the  relative  location  (range 
and  azunuth)  of  a  system  data  point. 

The  Anthro-Centric  Multisensory  Interface  for 
Sensory  Augmentation  of  Tele-Surgery  (ACMI- 
SATS)  project  seeks  to  empower  telerobotic 
surgery  with  many  of  the  sensory  and 
kinesthetic  cues  available  in  open  surgical 
procedures.  This  improves  the  effectiveness  of 
tele-surgery  by  allowing  a  surgeon  to  utilize 
tactile,  spatial  audio  and  three-dimensional 
visual  cues  as  well  as  meaningful  proprioceptive 
and  kinesthetic  information.  With  improved  SA, 
a  specialty  surgeon  could  perform  procedures  in 
multiple  cases  with  little  delay  between  them. 
Likewise,  multiple  surgeons  could  collaboratively 
control  the  same  or  multiple  robots  as  a  team. 


Description  of  the  Methods 

The  Florida  Institute  for  Human  and  Machine 
Cognition  (IHMC)  developed  the  ACMI-SATS 
system  with  both  simulated  and  actual  surgical 
robot  systems.  It  provides  an  integrated 
architecture  with  a  wide  Field  of  view,  pseudo- 
three  dimensional  visual  operative  field, 
surround  sound  for  spatially  relevant  audio 
when  selecting  instruments  and  multiple  tactile 
interfaces  that  represent  instrument  dynamics. 
ACMI-SATS  provides  a  natural,  free  motion 
control  interface  that  uses  a  wearable,  wireless 
motion-capture  system  to  track  head,  torso, 
arm,  wrist  and  finger  movements.  Natural 
motions  drive  the  movement  of  the  robotic 
instruments  and  the  endoscope.  The  surgeon 
can  stand  or  sit  without  any  external  restrictions 
and  proprioceptive  and  kinesthetic  sensations 
map  to  the  visual  presentation  at  a  1:1  scale. 
Motion  capture  enables  gestural  control  to 
manage  mode  changes  (such  as  clutching  or 
switching  control  to  a  different  arm). 

Preliminary  Results 

Initial  evaluations  on  the  ACMI-SATS  test  bed 
indicate  that  novices  (non-surgeons)  can  leam 
to  control  and  manipulate  the  robotic  end 
effectors  to  perform  minimally  invasive  surgical 
training  tasks  in  simulation  quickly  and  with  low 
cognitive  effort.  This  can  allow  a  single  surgeon 
to  quickly  and  safely  switch  between  control  of 
multiple,  non-identical  surgical  robots.  Model- 
based  smoothing  of  command  signals  provides 
robustness  to  the  variations  and  short  disruption 
inherent  to  long  distance  communications. 


Figure  1:  ACMI-SATS  smoothing  algorithm  ensures  reliability 
over  distance  of  robot  end  effector  dynamic  movements 
despite  data  or  network  communications  rate  variations. 
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(dV-tramer,  mimic  Technologies,  Inc.,  Seattle,  WA)  demon¬ 
strating  head  and  full  body  motion  capture  integrated  with 
auditory  and  tactile  information  displays  (inset  shows 
graphical  representation  of  manipulator  shafts  and  grasper 
angle;  each  also  indicates  the  forces  on  the  end  effectors). 


The  ACMI-SATS  sensory  interfaces  augment  SA 
by  providing  additional  information  to  the 
surgeon  intuitively,  without  overloading  visual  or 
auditory  sensory  channels.  The  gestural 
interface  allows  more  natural,  unrestricted 
movements  to  control  robot  actions  and  mode 
changes.  By  scaling  the  visual  display  to  human 
size,  the  surgeon  can  exploit  kinesthetic 
awareness  of  instrument  positions,  even  when 
they  are  no  longer  within  the  endoscope  field  of 
view.  Any  surgical  robot  platform  could  Integrate 
ACMI-SATS  and  allow  surgeons  to  leam,  as  well 
as  perform  more  procedures  in  a  given  time 
frame,  with  less  effort  and  fewer  surgical  errors. 
In  a  mass  casualty  event,  ACMI-SATS  could 
allow  teams  of  specialists  to  provide  definitive 
care  rapidly  to  multiple  patients,  collaborating 
on  each  with  operating  room  teams  deployed 
locally  with  multiple  surgical  robots. 


Figure  4;  ACMI-SATS  use/  swtehing  between  procedures 
using  gestures.  Physical  orientation  determines  the  active 
robot;  SA  displays  and  motion  capture  data  switch  between 


under  user  command. 
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Appendix  2:  Acronyms 


3D  -  Three  Dimensional 

ACMI-SATS  -  Anthro-Centric  Multisensory  Interface  for  Sensory  Augmentation  of  Tele-Surgery 
ADC  -  Analog  to  Digital  Converter 
AMI  -  Adaptive  Multiagent  Integration 
BMI  -  Brain-Machine  Interface 

BORIS  -  Basic  Operational  Robotics  Instruction  System 

DAC  -  Digital  to  Analog  Converter 

DARPA  -  Defense  Advanced  Research  Projects  Agency 

DoD  -  Department  of  Defense 

ERF  -  Electro-rheological  fluid 

fMRI  -  Functional  Magnetic  Resonance  Imaging 

FSM  -  Finite  State  Machine 

GUI-  Graphical  User  Interface 

HAPsMRT  -  High  Altitude  Platforms  Mobile  Robotic  Tele-surgery 

HD  -  High  Definition 

HMD  -  Head  Mounted  Display 

HMI-  Human  Machine  Interface 

IED  -  Improvised  Explosive  Device 

IHMC  -  Institute  for  Human  and  Machine  Cognition 

MRI  -  Magnetic  Resonance  Imaging 

NASA  -  National  Aeronautics  Space  Administration 

SA  -  Situation  Awareness 

SAGES  -  Society  of  American  Gastrointestinal  and  Endoscopic  Surgeons 
SLAM  -  Simultaneous  Localization  and  Mapping 
SCS  -  Simulation  Construction  Set 
TTI  -  Tactile  Torso  Interface 

TATRC  -  U.  S.  Army  Telemedicine  and  Advanced  Technology  Research  Center 
TSAS  -  Tactile  Situation  Awareness  System 
UAV  -  Unmanned  Aerial  Vehicle 
VE  -  Virtual  Environment 
VR  -  Virtual  Reality 
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